（物体呼啸） (object whooshing)
- 分词器的 pipeline 。 *[译者注: token, tokenization, tokenizer 等词均译成了 分词*, 实则不翻译最佳] - The tokenizer pipeline.
在本视频中，我们将了解分词器如何转换 In this video, we'll look at how a tokenizer converts
从原始文本到数字， raw texts to numbers,
Transformer 模型可以理解成， that a Transformer model can make sense of,
就像我们执行这段代码时一样。 like when we execute this code.
这是一个快速概述 Here is a quick overview
对于 tokenizer 对象内部发生的事情： of what happens inside the tokenizer object:
首先，文本被分成分词， first, the text is split into tokens,
它们是单词、单词的一部分或标点符号。 which are words, parts of words, or punctuation symbols.
然后分词器添加潜在的特殊分词 Then the tokenizer adds potential special tokens
并将每个分词转换为各自唯一的 ID and converts each token to their unique respective ID
由分词器的词汇表定义。 as defined by the tokenizer's vocabulary.
正如我们将要看到的，它并不完全是按照这个顺序发生的， As we'll see, it doesn't quite happen in this order,
但这样更利于理解。 but doing it like this is better for understandings.
第一步是将我们的输入文本拆分为分词。 The first step is to split our input text into tokens.
为此，我们使用 tokenize 方法。 We use the tokenize method for this.
为此，分词器可能首先执行一些操作， To do that, the tokenizer may first perform some operations,
比如将所有单词小写，然后遵循一组规则 like lowercasing all words, then follow a set of rules
将结果分成小块文本。 to split the result in small chunks of text.
大多数 Transformer 模型使用 Most of the Transformer models uses
单词分词化算法，这意味着 a word tokenization algorithm, which means
一个给定的单词可以被拆分 that one given word can be split
在几个分词中，例如这里的分词。 in several tokens like tokenize here.
查看下面的 “分词化算法” 视频链接 Look at the "Tokenization algorithms" video link below
了解更多信息。 for more information.
我们在 ize 前面看到的 ## 前缀是 The ## prefix we see in front of ize is
BERT 的约定, 用来表示 a convention used by BERT to indicate
这个分词不是单词的开头。 this token is not the beginning of the word.
然而，其他分词器可能使用不同的约定： Other tokenizers may use different conventions however:
例如，ALBERT 分词器会添加一个长下划线 for instance, ALBERT tokenizers will add a long underscore
在所有分词前 in front of all the tokens
在他们之前增加了空间，这是一个共同的约定 that added space before them, which is a convention shared
由所有句子分词器。 by all sentencepiece tokenizers.
分词化管道的第二步是 The second step of the tokenization pipeline is
将这些分词映射到它们各自的 ID to map those tokens to their respective IDs
由分词器的词汇定义的。 as defined by the vocabulary of the tokenizer.
这就是我们需要下载文件的原因 This is why we need to download the file
当我们实例化一个分词器时 when we instantiate a tokenizer
使用 from_pretrained 方法。 with the from_pretrained method.
我们必须确保我们使用相同的映射 We have to make sure we use the same mapping
和模型被预训练时一致。 as when the model was pretrained.
为此，我们使用 convert_tokens_to_ids 方法。 To do this, we use the convert_tokens_to_ids method.
我们可能已经注意到 We may have noticed
我们没有完全相同的结果 that we don't have the exact same results
就像我们的第一张幻灯片一样, 或许 as in our first slide, or not
因为这看起来像是一个随机数列表， as this looks like a list of random numbers anyway,
在这种情况下，请允许我重温一下你的回忆。 in which case, allow me to refresh your memory.
我们有缺少开头的数字和一个结尾的数字 We had a the number at the beginning and a number at the end, that are missing,
是特殊标记。 those are the special tokens.
特殊标记由 prepare_for_model 方法添加 The special tokens are added by the prepare_for_model method
它知道这个分词的索引 which knows the indices of this token
在词汇表中, 并仅添加适当的数字。 in the vocabulary and just adds the proper numbers.
在输入 ID 列表中。 in the input IDs list.
你可以查看特殊标记 You can look at the special tokens
更一般地说， and, more generally,
关于分词器如何更改你的文本， at how the tokenizer has changed your text,
通过使用 decode 方法 by using the decode method
在分词器对象的输出上。 on the outputs of the tokenizer object.
至于开头的前缀 As for the prefix for beginning
词的/词的部分，对于特殊标记, 依赖于 of words/ part of words, for special tokens vary depending
你正在使用哪个分词器。 on which tokenizer you're using.
因此分词器使用 CLS 和 SEP， So that tokenizer uses CLS and SEP,
但是 RoBERTa 分词器使用类似 HTML 的锚点 but the RoBERTa tokenizer uses HTML-like anchors
和 and .
既然你知道分词器是如何工作的， Now that you know how the tokenizer works,
你可以忘记所有那些中间方法， you can forget all those intermediate methods,
然后你记得你只需要调用它 and then you remember that you just have to call it
在你的输入文本上。 on your input texts.
分词器的输出不只是 The output of a tokenizer don't
仅包含输入 ID, 然而. just contain the input IDs, however.
要了解注意力掩码是什么， To learn what the attention mask is,
查看 “一起批量输入” 视频。 check out the "Batch input together" video.
要了解分词类型 ID， To learn about token type IDs,
观看 “处理句子对” 视频。 look at the "Process pairs of sentences" video.
（物体呼啸） (object whooshing)