def get_tokens_as_list()

in distilvit/update_config.py [0:0]

• 6 lines of code
• 2 McCabe index (conditional complexity)

def get_tokens_as_list(word_list):
    tokens_list = []
    for word in word_list:
        tokenized_word = tokenizer([word], add_special_tokens=False).input_ids[0]
        tokens_list.append(tokenized_word)
    return tokens_list