（空气呼啸） (air whooshing)
在本视频中，我们将看到如何 In this video, we will see how
你可以从头开始创建自己的 tokenizer(分词器) 。 you can create your own tokenizer from scratch.
要创建自己的 tokenizer ，你必须考虑 To create your own tokenizer, you will have to think about
分词化中涉及的每个操作。 each of the operations involved in tokenization.
即，规范化，预标记化， Namely, the normalization, the pre-tokenization,
模型、后处理和解码。 the model, the post processing, and the decoding.
如果你不知道什么是规范化， If you don't know what normalization,
预标记化，模型是， pre-tokenization, and the model are,
我建议你去看下面链接的视频。 I advise you to go and see the videos linked below.
后处理包括所有修改 The post processing gathers all the modifications
我们将对分词化文本执行的。 that we will carry out on the tokenized text.
它可以包括添加特殊 token ， It can include the addition of special tokens,
创建一个意图掩码， the creation of an intention mask,
还会生成 token 的 ID 列表。 but also the generation of a list of token IDs.
解码操作发生在最后， The decoding operation occurs at the very end,
并将允许通过 and will allow passing
来自句子中的 ID 序列。 from the sequence of IDs in a sentence.
例如，你可以看到 hash 标签 For example, you can see that the hashtags
已被删除，并且 token have been removed, and the tokens
今天的词都归在了一起。 composing the word today have been grouped together.
在快速 tokenizer 中，所有这些组件 In a fast tokenizer, all these components
收集在 backend_tokenizer 属性中。 are gathered in the backend_tokenizer attribute.
正如你在这个小片代码中看到的那样， As you can see with this small code snippet,
它是 tokenizer 的一个实例 it is an instance of a tokenizer
来自 tokenizers 库。 from the tokenizers library.
因此，要创建自己的 tokenizer ， So, to create your own tokenizer,
你将必须遵循这些步骤。 you will have to follow these steps.
第一，创建一个训练数据集。 First, create a training dataset.
第二, 创建和训练 tokenizer Second, create and train a tokenizer
用 transformer 库。 with the transformer library.
第三，将此 tokenizer 加载为 transformer 的 tokenizer 中。 And third, load this tokenizer into a transformer tokenizer.
要了解这些步骤， To understand these steps,
我建议我们一起重新创建一个 BERT 分词器。 I propose that we recreate a BERT tokenizer together.
首先要做的是创建一个数据集。 The first thing to do is to create a dataset.
使用此代码片段，你可以创建一个迭代器 With this code snippet you can create an iterator
在数据集 wikitext-2-raw-V1 上， on the dataset wikitext-2-raw-V1,
这是一个相当小的英语数据集， which is a rather small dataset in English,
完美的例子。 perfect for the example.
我们主要修改这里， We attack here the big part,
使用 tokenizer 库设计我们的 tokenizer 。 the design of our tokenizer with the tokenizer library.
我们首先初始化一个 tokenizer 实例 We start by initializing a tokenizer instance
使用 WordPiece 模型，因为它是 BERT 使用的模型。 with a WordPiece model because it is the model used by BERT.
然后我们可以定义我们的规范化器。 Then we can define our normalizer.
我们将其定义为连续的两个规范化 We will define it as a succession of two normalizations
用于清理文本中不可见的字符。 used to clean up characters not visible in the text.
一个小写归一化， One lowercasing normalization,
最后两个标准化用于删除重音。 and two last normalizations used to remove accents.
对于预标记化，我们将链接两个 pre_tokenizers。 For the pre-tokenization, we will chain two pre_tokenizers.
第一个在空格级别分隔文本， The first one separating the text at the level of spaces,
第二个隔离标点符号。 and the second one isolating the punctuation marks.
现在，我们可以定义允许我们的训练方法 Now, we can define the trainer that will allow us
训练一开始选择的 WordPiece 模型。 to train the WordPiece model chosen at the beginning.
为了开展训练， To carry out the training,
我们将不得不选择词汇量。 we will have to choose a vocabulary size.
这里我们选择 25,000。 Here we choose 25,000.
我们还需要公布特殊 token And we also need to announce the special tokens
我们绝对想添加到我们的词汇表中。 that we absolutely want to add to our vocabulary.
在一行代码中，我们可以训练我们的 WordPiece 模型 In one line of code, we can train our WordPiece model
使用我们之前定义的迭代器。 using the iterator we defined earlier.
模型训练完成后，我们可以检索 Once the model has been trained, we can retrieve
特殊类别和分离 token 的 ID， the IDs of the special class and separation tokens,
因为我们需要它们来对我们的序列进行后期处理。 because we will need them to post-process our sequence.
感谢 TemplateProcessing 类， Thanks to the TemplateProcessing class,
我们可以在每个序列的开头添加 CLS token ， we can add the CLS token at the beginning of each sequence,
和序列末尾的 SEP token ， and the SEP token at the end of the sequence,
如果我们标记一对文本，则在两个句子之间。 and between two sentences if we tokenize a pair of text.
最后，我们只需要定义我们的解码器， Finally, we just have to define our decoder,
这将允许我们删除主题标签 which will allow us to remove the hashtags
在 token 的开头 at the beginning of the tokens
必须重新附加到以前的 token 。 that must be reattached to the previous token.
它就在那里。 And there it is.
你拥有所有必要的代码行 You have all the necessary lines of code
使用 tokenizer 库定义你自己的 tokenizer 。 to define your own tokenizer with the tokenizer library.
现在我们有了一个全新的 tokenizer Now that we have a brand new tokenizer
使用 tokenizer 库，我们只需要加载它 with the tokenizer library, we just have to load it
从 transformers 库中转换为快速 tokenizer 。 into a fast tokenizer from the transformers library.
同样，我们有几种可能性。 Here again, we have several possibilities.
我们可以在一般类中加载它， We can load it in the generic class,
PreTrainedTokenizerFast，或在 BertTokenizerFast 类中 PreTrainedTokenizerFast, or in the BertTokenizerFast class
因为我们在这里构建了一个类似 BERT 的分词器。 since we have built a BERT like tokenizer here.
我真的希望这个视频能帮助你理解 I really hope this video has helped you understand
如何创建自己的 tokenizer ， how you can create your own tokenizer,
并且你现在已准备好导航 and that you are ready now to navigate
tokenizer 库文档 the tokenizer library documentation
为你的全新 tokenizer 选择组件。 to choose the components for your brand new tokenizer.
（空气呼啸） (air whooshing)