（空气呼啸） (air whooshing)
- 你是在正确的地方 - You are at the right place
如果你想了解什么是字节对编码(BPE)算法 if you want to understand what the Byte Pair Encoding
子词分词化算法是， subword tokenization algorithm is,
如何训练它 how to train it
以及文本的分词化是如何完成的 and how the tokenization of a text is done
用这个算法。 with this algorithm.
BPE 算法 The BPE algorithm
最初被提出作为文本压缩算法 was initially proposed as a text compression algorithm
但它也非常适合 but it is also very well suited
作为你的语言模型的分词器。 as a tokenizer for your language models.
BPE 的思想是分词 The idea of BPE is to divide words
进入一系列 “子词单元” into a sequence of'subword units'
其是参考语料库中频繁出现的单位 which are units that appear frequently in a reference corpus
也是我们用来训练它的语料库。 which is, the corpus we used to train it.
BPE 分词器是如何训练的？ How is a BPE tokenizer trained?
首先，我们必须得到一个文本语料库。 First of all, we have to get a corpus of texts.
我们不会在这个原始文本上训练我们的分词器 We will not train our tokenizer on this raw text
但我们首先将其规范化 but we will first normalize it
然后对其进行预标记。 then pre-tokenize it.
作为预标记化 As the pre-tokenization
将文本分成单词列表， divides the text into a list of words,
我们可以用另一种方式表示我们的语料库 we can represent our corpus in another way
通过收集相同的词 by gathering together the same words
并通过维护一个柜， and by maintaining a counter,
这里用蓝色表示。 here represented in blue.
要了解训练的工作原理， To understand how the training works,
我们认为这个小语料库由以下单词组成： we consider this toy corpus composed of the following words:
huggingface, hugging, hug, hugger, 等 huggingface, hugging, hug, hugger, etc.
BPE 是一种从初始词汇表开始的算法 BPE is an algorithm that starts with an initial vocabulary
然后将其增加到所需的大小。 and then increases it to the desired size.
为了建立初始词汇表， To build the initial vocabulary,
我们从分离语料库的每个词开始 we start by separating each word of the corpus
组成它们的基本单元列表， into a list of elementary units that compose them,
在这里, 字符。 here, the characters.
我们在词汇表中列出所有出现的字符 We list in our vocabulary all the characters that appear
这将构成我们最初的词汇表。 and that will constitute our initial vocabulary.
现在让我们看看如何增加它。 Let's now see how to increase it.
我们回到我们拆分的语料库， We return to our split corpus,
我们将逐字逐句 we will go through the words one by one
并计算 token 对的所有出现次数。 and count all the occurrences of token pairs.
第一对由标记 “h” 和 “u” 组成， The first pair is composed of the token 'h' and 'u',
第二个 "u" 和 "g"， the second 'u' and 'g',
我们继续这样，直到我们有完整的列表。 and we continue like that until we have the complete list.
一旦我们知道所有的对 Once we know all the pairs
以及它们出现的频率， and their frequency of appearance,
我们将选择出现频率最高的那个。 we will choose the one that appears the most frequently.
这是由字母 “l” 和 “e” 组成的一对。 Here it is the pair composed of the letters 'l' and 'e'.
我们注意到我们的第一个合并规则 We note our first merging rule
然后我们将新 token 添加到我们的词汇表中。 and we add the new token to our vocabulary.
然后我们可以将此合并规则应用于我们的拆分。 We can then apply this merging rule to our splits.
你可以看到我们已经合并了所有的 token 对 You can see that we have merged all the pairs of tokens
由标记 “l” 和 “e” 组成。 composed of the tokens 'l' and 'e'.
现在，我们只需要重现相同的步骤 And now, we just have to reproduce the same steps
与我们的新拆分。 with our new splits.
我们计算出现频率 We calculate the frequency of occurrence
对每对 token ， of each pair of tokens,
我们选择频率最高的一对， we select the pair with the highest frequency,
我们在我们的合并规则中注意到它， we note it in our merge rules,
我们将新的 token 添加到词汇表中 we add the new one token the vocabulary
然后我们合并所有的 token 对 and then we merge all the pairs of tokens
由标记 “le” 和 “a” 组成，进入我们的拆分。 composed of the token 'le' and 'a' into our splits.
我们可以重复这个操作 And we can repeat this operation
直到我们达到所需的词汇量。 until we reach the desired vocabulary size.
在这里，当我们的词汇量达到 21 个 token 时，我们就停止了。 Here, we stopped when our vocabulary reached 21 tokens.
我们现在可以看到，比起训练开始时 We can see now that the words of our corpus
我们的语料库中的单词 are now divided into far fewer tokens
被分成了更少的 token than at the beginning of the training.
而我们的算法 And that our algorithm
学会了部首 “hug” 和 “learn” has learned the radicals 'hug' and 'learn'
以及动词结尾 “ing”。 and also the verbal ending 'ing'.
现在我们已经学会了我们的词汇 Now that we have learned our vocabulary
和合并规则，我们可以标记新文本。 and merging rules, we can tokenize new texts.
例如， For example,
如果我们想标记 “hugs” 这个词， if we want to tokenize the word 'hugs',
首先我们将它分成基本单元 first we'll divide it into elementary units
所以它变成了一个字符序列。 so it became a sequence of characters.
然后，我们将通过我们的合并规则 Then, we'll go through our merge rules
直到我们有一个我们可以应用。 until we have one we can apply.
在这里，我们可以合并字母 “h” 和 “u”。 Here, we can merge the letters 'h' and 'u'.
在这里，我们可以合并 2 个 token 以获得新 token “hug”。 And here, we can merge 2 tokens to get the new token 'hug'.
当我们到达合并规则的末尾时， When we get to the end of our merge rules,
分词化完成。 the tokenization is finished.
就是这样。 And that's it.
我希望现在的 BPE 算法 I hope that now the BPE algorithm
对你而言不再是秘密！ has no more secret for you!
（空气呼啸） (air whooshing)