- 分词化管线涉及几个步骤 - The tokenization pipeline involves several steps
将原始文本转换为数字。 that converts raw text into numbers.
在这段视频中，我们将看到会发生什么 In this video, we will see what happens
在预分词化步骤中。 during the pre-tokenization step.
预分词化操作是操作执行在 The pre-tokenization operation is the operation performed
文本规范化后 after the normalization of the text
在应用分词化算法之前。 and before the application of the tokenization algorithm.
此步骤包括应用规则 This step consists in applying rules
不需要学习的 that do not need to be learned
执行文本的第一部分。 to perform a first division of the text.
让我们看看几个分词器是如何 Let's look at how several tokenizers
在此示例中进行预分词。 pre-tokenize in this example.
gpt2 pre-tokenization 把文本划分到空格 The gpt2 pre-tokenization divides the text on spaces
和一些标点符号，但不是撇号。 and some punctuation, but not on the apostrophe.
我们还注意到空格已被替换 We also notice that spaces have been replaced
上面有一个点的大写字母 G。 by capital G with a dot above.
Albert 的预分词化将文本划分 Albert's pre-tokenization divides the text
在空间层面， at the level of spaces,
在句子的开头加一个空格， adds a space at the beginning of the sentence,
并用特殊下划线替换空格。 and replaces spaces with a special underscore.
最后，Bert 的 pre-tokenization 对文本进行分词 Finally, Bert's pre-tokenization divides the text
在标点符号和空格的级别。 at the level of punctuation and spaces.
但与之前的分词器不同， But unlike the previous tokenizers,
空格没有变换 spaces are not transformed
并集成到使用此预标记器生成的 token 中。 and integrated into tokens produced with this pre-tokenizer.
通过这三个例子， Through this three example,
我们可以观察到两种主要的操作类型 we could observe the two main type of operation
预分词化带来的； brought by the pre-tokenization;
对案文进行一些改动 some change on the text
并将字符串划分为标记 and the division of the string into tokens
可以与单词相关联。 that can be associated to words.
最后，快速分词器的后端分词器 Finally, the backend tokenizer of the fast tokenizers
还允许测试预分词化操作 also allows to test the pre-tokenization operation
非常容易，这要归功于它的 pre_tokenize_str 方法。 very easily, thanks to its pre_tokenize_str method.
我们注意到这个操作的输出 We notice that the output of this operation
由 token 和偏移量组成， is composed of both tokens and offsets,
允许将 token 链接到它在文本中的位置 which allow to link the tokens to its position in the text
在方法的输入中给出。 given in input of the method.
此操作定义了最大的 token This operation defines the largest tokens
可以通过分词化产生， that can be produced by the tokenization,
或者换句话说，子 token 的障碍 or in those words, the barriers of the sub-tokens
届时将产生。 which will be produced then.
这就是特性的全部 And that's all for the characteristic
对预分词器。 of the pre-tokenizers.