（空气呼啸） (air whooshing)
- 一起来看看什么是训练策略 - Let's see together what is the training strategy
对 WordPiece 算法及其执行方式 of the WordPiece algorithm, and how it performs
一旦训练，文本的分词化。 the tokenization of a text, once trained.
WordPiece 是 Google 推出的一种分词算法。 WordPiece is a tokenization algorithm introduced by Google.
例如，它被 BERT 使用。 It is used, for example, by BERT.
据我们所知， To our knowledge,
WordPiece 的代码还没有开源。 the code of WordPiece has not been open source.
所以我们根据我们的解释 So we base our explanations
根据我们自己对已发表文献的理解。 on our own interpretation of the published literature.
那么，WordPiece 的训练策略是怎样的呢？ So, what is the training strategy of WordPiece?
与 BPE 算法类似， Similarly to the BPE algorithm,
WordPiece 从建立初始词汇表开始 WordPiece starts by establishing an initial vocabulary
由基本单元组成， composed of elementary units,
然后将这个词汇量增加到所需的大小。 and then increases this vocabulary to the desired size.
为了建立初始词汇表， To build the initial vocabulary,
我们划分训练语料库中的每个单词 we divide each word in the training corpus
进入构成它的字母序列。 into the sequence of letters that make it up.
如你所见，有一个小细节。 As you can see, there is a small subtlety.
我们在字母前添加两个标签 We add two hashtags in front of the letters
那不开启一个词。 that do not start a word.
通过每个基本单元只保留一次出现， By keeping only one occurrence per elementary unit,
我们现在有了最初的词汇表。 we now have our initial vocabulary.
我们将列出语料库中所有现有的对。 We will list all the existing pairs in our corpus.
一旦我们有了这份表格， Once we have this list,
我们将为每一对计算一个分数。 we will calculate a score for each of these pairs.
至于 BPE 算法， As for the BPE algorithm,
我们将选择得分最高的一对。 we will select the pair with the highest score.
举个例子， Taking for example,
第一对由字母 H 和 U 组成。 the first pair composed of the letters H and U.
一对的分数单纯地等于频率 The score of a pair is simply equal to the frequency
对这对出现的, 除以乘积 of appearance of the pair, divided by the product
对第一个 token 出现的频率， of the frequency of appearance of the first token,
乘上第二个 token 的出现频率。 by the frequency of appearance of the second token.
因此，在一对固定的出现频率下， Thus, at a fixed frequency of appearance of the pair,
如果该对的子部分在语料库中非常频繁， if the subparts of the pair are very frequent in the corpus,
那么这个分数就会降低。 then this score will be decreased.
在我们的示例中，HU 对出现了四次， In our example, the pair HU appears four times,
字母 H 四次，字母 U 四次。 the letter H four times, and the letter U four times.
这给了我们 0.25 的分数。 This gives us a score of 0.25.
现在我们知道如何计算这个分数了， Now that we know how to calculate this score,
我们可以为所有对做。 we can do it for all pairs.
我们现在可以添加到词汇表中 We can now add to the vocabulary
得分最高的一对，当然是在合并之后。 the pair with the highest score, after merging it of course.
现在我们可以将同样的操作应用于我们的拆分语料库。 And now we can apply this same fusion to our split corpus.
你可以想象， As you can imagine,
我们只需要重复相同的操作 we just have to repeat the same operations
直到我们拥有所需大小的词汇表。 until we have the vocabulary at the desired size.
让我们再看几个步骤 Let's look at a few more steps
看看我们词汇的演变， to see the evolution of our vocabulary,
以及拆分长度的演变。 and also the evolution of the length of the splits.
现在我们对自己的词汇感到满意了， And now that we are happy with our vocabulary,
你可能想知道如何使用它来标记文本。 you are probably wondering how to use it to tokenize a text.
假设我们想要标记 “huggingface” 这个词。 Let's say we want to tokenize the word "huggingface".
WordPiece 遵循这些规则。 WordPiece follows these rules.
我们将寻找最长的 token We will look for the longest possible token
在单词的开头。 at the beginning of the word.
然后我们重新开始我们的词语的剩余部分， Then we start again on the remaining part of our word,
依此类推，直到我们到达终点。 and so on until we reach the end.
就是这样。 Huggingface 分为四个子 token 。 And that's it. Huggingface is divided into four sub-tokens.
本视频即将结束。 This video is about to end.
我希望它能帮助你更好地理解 I hope it helped you to understand better
工作的背后是什么，WordPiece。 what is behind the work, WordPiece.
（空气呼啸） (air whooshing)