def get_tokenizer()

in src/alignment/model_utils.py [0:0]

• 2 lines of code
• 1 McCabe index (conditional complexity)

def get_tokenizer(
    model_args: ModelArguments, data_args: DataArguments, auto_set_chat_template: bool = True