（空气呼啸） (air whooshing)
- 在这段视频中，我们将一起看到 - In this video we will see together
训练 tokenizer 的目的是什么， what is the purpose of training a tokenizer,
要遵循的关键步骤是什么， what are the key steps to follow,
什么是最简单的方法。 and what is the easiest way to do it.
你会问自己这个问题， You will ask yourself the question,
“我应该训练一个新的 tokenizer 吗？”， "Should I train a new tokenizer?",
当你计划从头开始训练新模型时。 when you plan to train a new model from scratch.
一个训练过的分词器会不适合你的语料库 A trained tokenizer would not be suitable for your corpus
如果你的语料库使用一个不同的语言， if your corpus is in a different language,
使用新字符，比如重音符号或大写字母， uses new characters, such as accents or upper cased letters,
有特定的词汇，例如医学的或法律的， has a specific vocabulary, for example medical or legal,
或使用一个不同的风格， or uses a different style,
例如，来自另一个世纪时的语言。 a language from another century for example.
如果我让 tokenizer 训练在 If I take the tokenizer trained on
bert-base-uncased 模型， the bert-base-uncased model,
并忽略其归一化步骤， and ignore its normalization step,
然后我们就能看到 tokenization 操作 then we can see that the tokenization operation
在英语句子上， on the English sentence,
“这是一个适合我们 tokenizer 的句子”， "Here is a sentence adapted to our tokenizer",
产生一个相当令人满意的 token 列表， produces a rather satisfactory list of tokens,
从某种意义上说，这八个字的句子 in the sense that this sentence of eight words
被标记成了九个 token 。 is tokenized into nine tokens.
另一方面，如果我使用相同的 tokenizer On the other hand, if I use this same tokenizer
在孟加拉语的一个句子中，我们看到 on a sentence in Bengali, we see that
要么一个词被分成许多子 token ， either a word is divided into many sub tokens,
或者 tokenizer 不知道任何一个 or that the tokenizer does not know one of
unicode 字符并仅返回未知 token 。 the unicode characters and returns only an unknown token.
一个常用词被分成许多 token 的事实 The fact that a common word is split into many tokens
可能是问题的，因为语言模型 can be problematic, because language models
只能处理有限长度的 token 序列。 can only handle a sequence of tokens of limited length.
过度拆分初始文本的 tokenizer A tokenizer that excessively splits your initial text
甚至可能影响模型的性能。 may even impact the performance of your model.
未知的 token 也有问题， Unknown tokens are also problematic,
因为模型将无法提取 because the model will not be able to extract
来自文本未知部分的任何信息。 any information from the unknown part of the text.
在另一个例子中，我们可以看到 In this other example, we can see that
tokenizer 替换包含字符的单词 the tokenizer replaces words containing characters
带有重音和带有未知 token 的大写字母。 with accents and capital letters with unknown tokens.
最后，如果我们再次使用这个 tokenizer Finally, if we use again this tokenizer
为了标记医学词汇，我们再次看到 to tokenize medical vocabulary, we see again that
一个单词被分成许多子 token ， a single word is divided into many sub tokens,
四个用于 "paracetamol" ，四个用于 "pharyngitis" 。 four for paracetamol, and four for pharyngitis.
目前的大多数 tokenizer 被 Most of the tokenizers used by the current
最先进的语言模型使用的, 需要被训练 state of the art language models need to be trained
在语料库上, 其与使用的语料库如要相似于 on a corpus that is similar to the one used
预训练语言模型的。 to pre-train the language model.
这种训练包括学习规则 This training consists in learning rules
将文本分成标记。 to divide the text into tokens.
以及学习和使用这些规则的方法 And the way to learn these rules and use them
取决于所选的 tokenizer 模型。 depends on the chosen tokenizer model.
因此，要训练一个新的 tokenizer ，首先需要 Thus, to train a new tokenizer, it is first necessary
构建由原始文本组成的训练语料库。 to build a training corpus composed of raw texts.
然后，你必须为你的 tokenizer 选择一种结构。 Then, you have to choose an architecture for your tokenizer.
这里有两个选项。 Here there are two options.
最简单的是重用与那个结构, 其相同于 The simplest is to reuse the same architecture as the one
另一个已经训练过的模型使用的 tokenizer 。 of a tokenizer used by another model already trained.
否则也可以 Otherwise it is also possible
彻底设计你的分词器。 to completely design your tokenizer.
但这需要更多的经验和观察。 But it requires more experience and attention.
一旦选择了结构，你就可以 Once the architecture is chosen, you can thus
在你构建的语料库上训练这个 tokenizer 。 train this tokenizer on your constituted corpus.
最后，你需要做的最后一件事就是保存 Finally, the last thing that you need to do is to save
能够使用此 tokenizer 的学习规则。 the learned rules to be able to use this tokenizer.
让我们举个例子。 Let's take an example.
假设你想在 Python 代码上训练 GPT-2 模型。 Let's say you want to train a GPT-2 model on Python code.
即使 Python 代码通常是英文的 Even if the Python code is usually in English
这种类型的文本非常具体， this type of text is very specific,
并值得一个训练过的 tokenizer 。 and deserves a tokenizer trained on it.
为了让你相信这一点，我们将在最后看到 To convince you of this, we will see at the end
一个例子产生的差异。 the difference produced on an example.
为此，我们将使用该方法 For that we are going to use the method
“train_new_from_iterator”，所有快速的 tokenizer "train_new_from_iterator" that all the fast tokenizers
在库中有的，因此， of the library have and thus,
特别是 GPT2TokenizerFast。 in particular GPT2TokenizerFast.
这是我们案例中最简单的方法 This is the simplest method in our case
有一个适合 Python 代码的 tokenizer 。 to have a tokenizer adapted to Python code.
请记住，第一件事是收集训练语料库。 Remember, the first thing is to gather a training corpus.
我们将使用 CodeSearchNet 数据集的一个子部分 We will use a subpart of the CodeSearchNet dataset
仅包含 Python 函数 containing only Python functions
来自 Github 上的开源库。 from open source libraries on Github.
这是个好时机。 It's good timing.
该数据集为数据集库所知 This dataset is known by the datasets library
我们可以用两行代码加载它。 and we can load it in two lines of code.
然后，正如 “train_new_from_iterator” 方法所期望的那样 Then, as the "train_new_from_iterator" method expects
文本列表的迭代器， a iterator of lists of texts,
我们创建 “get_training_corpus” 函数， we create the "get_training_corpus" function,
这将返回一个迭代器。 which will return an iterator.
现在我们有了迭代器 Now that we have our iterator
在我们的 Python 函数语料库中，我们可以加载 on our Python functions corpus, we can load
GPT-2 分词器结构。 the GPT-2 tokenizer architecture.
这里 old_tokenizer 不适合我们的语料库。 Here old_tokenizer is not adapted to our corpus.
但我们只需要 But we only need
再用一行在我们的新语料库上训练它。 one more line to train it on our new corpus.
有一点很普遍的是大多数 分词化 An argument that is common to most of the tokenization
算法, 目前使用的是根据词汇表的大小。 algorithms used at the moment is the size of the vocabulary.
我们在这里选择值 52,000。 We choose here the value 52,000.
最后，一旦训练结束， Finally, once the training is finished,
我们只需要在本地保存我们的新 tokenizer ， we just have to save our new tokenizer locally,
或将其发送到 hub 以便能够重用它 or send it to the hub to be able to reuse it
之后很容易。 very easily afterwards.
最后大家一起看个例子有没有用 Finally, let's see together on an example if it was useful
重新训练一个类似于 GPT-2 的 tokenizer 。 to re-train a tokenizer similar to GPT-2 one.
使用 GPT-2 的原始 tokenizer With the original tokenizer of GPT-2
我们看到所有的空间都是孤立的， we see that all spaces are isolated,
和方法名称 "randn"， and the method name randn,
在 Python 代码中相对比较常见，分为两部分。 relatively common in Python code, is split in two.
使用我们新的分词器，单缩进和双缩进 With our new tokenizer, single and double indentations
已经学习和方法 "randn" have been learned and the method randn
被分词化为一个 token 。 is tokenized into one token.
有了这个， And with that,
你现在知道如何训练你自己的分词器了。 you now know how to train your very own tokenizers now.
（空气呼啸） (air whooshing)