（空气呼啸） (air whooshing)
- 在这个视频中， - In this video,
我们将一起研究 “Unigram 语言模型 we will study together 'the Unigram Language Model
子词分词化算法 " 。 subword tokenization algorithm'.
整体训练策略 The overall training strategy
对一个 Unigram 语言模型分词器 of a Unigram Language Model tokenizer
是从一个非常大的词汇量开始 is to start with a very large vocabulary
然后在每次迭代中删除 token and then to remove tokens at each iteration
直到我们达到所需的大小。 until we reach the desired size.
在每次迭代中， At each iteration,
我们将计算训练语料库的损失 we will calculate a loss on our training corpus
多亏了 Unigram 模型。 thanks to the Unigram model.
由于损失计算取决于可用的词汇表， As the loss calculation depends on the available vocabulary,
我们可以用它来选择如何减少词汇量。 we can use it to choose how to reduce the vocabulary.
所以我们看看损失的变化 So we look at the evolution of the loss
通过依次从词汇表中删除每个 token 。 by removing in turn each token from the vocabulary.
我们将选择删除 p 百分比 We will choose to remove the p-percents
增加的损失越少。 which increase the loss the less.
在进一步之前 Before going further
在训练算法的解释中， in the explanation of the training algorithm,
我需要解释什么是 Unigram 模型。 I need to explain what is an Unigram model.
Unigram 语言模型 The Unigram Language Model
是一种统计语言模型。 is a type of Statistical Language Model.
统计语言模型 A Statistical Language Model
将为文本分配概率 will assign a probability to a text
考虑到文本实际上是一系列 token 。 considering that the text is in fact a sequence of tokens.
可以想象的最简单的 token 序列 The simplest sequences of tokens to imagine
是组成句子或字符的单词。 are the words that compose the sentence or the characters.
Unigram 语言模型的特殊性 The particularity of Unigram Language Model
是它假设每个词的出现 is that it assumes that the occurrence of each word
独立于它的前一个词。 is independent of its previous word.
这个假设允许我们写 This assumption allows us to write
一个文本的概率 that the probability of a text
等于概率的乘积 is equal to the product of the probabilities
对组成它的 token 。 of the tokens that compose it.
这里需要注意的是，它是一个非常简单的模型 It should be noted here that it is a very simple model
这不会适应文本的生成 which would not be adapted to the generation of text
因为这个模型总是会生成相同的 token ， since this model would always generate the same token,
概率最大的那个。 the one which has the greatest probability.
然而，要进行分词化， Nevertheless, to do tokenization,
这个模型对我们很有用 this model is very useful to us
因为它可以使用 because it can be used
估计不同短语的相对可能性。 to estimate the relative likelihood of different phrases.
我们现在准备好了 We are now ready
回到我们对训练算法的解释。 to return to our explanation of the training algorithm.
假设我们有一个训练语料库 Let's say that we have as a training corpus
用 10 次 hug 这个词， with 10 times the word hug,
12 次 pug 这个词， 12 times the word pug,
5 次 lug 这个词， 5 times the word lug,
4 次 bug 4 times bug
5 次 dug。 and 5 times dug.
如前所述， As said earlier,
训练从大量词汇开始。 the training starts with a big vocabulary.
显然，由于我们使用的是玩具语料库， Obviously, as we are using a toy corpus,
这个词汇量不会那么大 this vocabulary will not be that big
但它应该告诉你原理。 but it should show you the principle.
第一种方法是列出所有可能的严格子串 A first method is to list all the possible strict substrings
这就是我们在这里要做的。 and that's what we'll do here.
我们也可以使用 BPE 算法 We could also have used the BPE algorithm
词汇量非常大 with a very large vocabulary size
但就目前而言，严格的子字符串就足够了。 but for now, the strict substrings are enough.
Unigram 分词器的训练 The training of the Unigram tokenizer
基于期望最大化方法。 is based on the Expectation-Maximization method.
在每次迭代中， At each iteration,
我们估计 token 的概率 we estimate the probabilities of the tokens
对词汇 of the vocabulary
然后我们删除 p 百分比的 token and then we remove the p-percent of tokens
最小化语料库的损失 that minimize the loss on the corpus
而哪些不属于基本字 and which do not belong to the basic character
因为我们想保留在我们的最终词汇表中 as we want to keep in our final vocabulary
能够标记任何单词的基本字符。 the basic characters to be able to tokenize any word.
让我们开始吧！ Let's go for it!
简单估计一个 token 的概率 The probability of a token simply estimated
按此 token 出现的次数 by the number of appearance of this token
在我们的训练语料库中 in our training corpus
除以所有 token 出现的总数。 divided by the total number of appearance of all the tokens.
我们可以使用这个词汇表来标记我们的单词 We could use this vocabulary to tokenize our words
根据 Unigram 模型。 according to the Unigram model.
我们将一起做，以了解两件事： We will do it together to understand two things:
我们如何使用 Unigram 模型对单词进行分词 how we tokenize a word with a Unigram model
以及如何在我们的语料库上计算损失。 and how the loss is calculated on our corpus.
我们的文本 “Hug” 的 Unigram LM 分词化 The Unigram LM tokenization of our text 'Hug'
将是发生概率最高的那个 will be the one with the highest probability of occurrence
根据我们的 Unigram 模型。 according to our Unigram model.
找到它，最简单的方法 To find it, the simplest way to proceed
将列出所有可能的分割 would be to list all the possible segmentations
对我们的文本 "Hug" , of our text 'Hug',
计算每个细分的概率 calculate the probability of each of these segmentations
然后选择概率最高的那个。 and then choose the one with the highest probability.
以现在的词汇量， With the current vocabulary,
两个分词化获得完全相同的概率。 two tokenizations get exactly the same probability.
所以我们选择其中之一 So we choose one of them
并记住相关的概率。 and keep in memory the associated probability.
为了计算我们训练语料库的损失， To compute the loss on our training corpus,
我们需要像刚才那样进行分词化 we need to tokenize as we just did
语料库中所有剩余的单词。 all the remaining words in the corpus.
损失就是所有单词的总和, 语料库中 The loss is then the sum over all the words in the corpus
词的出现频率 of the frequency of occurrence of the word
乘以概率对数的相反数 multiplied by the opposite of the log of the probability
与单词的分词化相关联的。 associated with the tokenization of the word.
我们在这里得到了 170 的损失。 We obtain here a loss of 170.
请记住，我们最初的目标是减少词汇量。 Remember, our initial goal was to reduce the vocabulary.
为此，我们将从词汇表中删除一个 token To do this, we will remove a token from the vocabulary
并计算相关损失。 and calculate the associated loss.
例如，让我们删除标记 “ug”。 Let's remove for example, the token 'ug'.
我们注意到 “hug” 的分词化 We notice that the tokenization for 'hug'
字母 "h" 和元组 "ug" 现在是不可能的。 with the letter 'h' and the tuple 'ug' is now impossible.
尽管如此，正如我们之前看到的 Nevertheless, as we saw earlier
两个分词化具有相同的概率， that two tokenizations had the same probability,
我们仍然可以选择剩余的分词化 we can still choose the remaining tokenization
概率为 1.10e-2。 with a probability of 1.10e-2.
词汇表中其他词的分词 The tokenizations of the other words of the vocabulary
也保持不变。 also remain unchanged.
最后，即使我们删除了标记 “ug” And finally, even if we remove the token 'ug'
从我们的词汇表来看，损失仍然等于 170。 from our vocabulary the loss remains equal to 170.
对于第一次迭代， For this first iteration,
如果我们继续计算， if we continue the calculation,
我们会注意到我们可以删除任何 token we would notice that we could remove any token
在不影响损失的情况下。 without it impacting the loss.
因此，我们将随机选择删除标记 “ug” We will therefore choose at random to remove the token 'ug'
在开始第二次迭代之前。 before starting a second iteration.
所以我们再次估计每个 token 的概率 So we estimate again the probability of each token
在计算每个 token 对损失的影响之前。 before calculating the impact of each token on the loss.
例如，如果我们现在删除 For example, if we remove now
由字母 “h” 和 “u” 组成的 token ， the token composed of the letters 'h' and 'u',
"hug" 只剩下一种可能的分词化。 there is only one possible tokenization left for "hug".
词汇表中其他词的分词化 The tokenization of the other words of the vocabulary
没有改变。 is not changed.
到底， In the end,
我们通过删除 token 获得 we obtain by removing the token
由词汇表中的字母 “h” 和 “u” 组成， composed of the letters 'h' and 'u' from the vocabulary,
损失为 168。 a loss of 168.
最后，要选择要删除的 token ， Finally, to choose which token to remove,
我们将为词汇表的每个剩余 token ， we will for each remaining token of the vocabulary,
这不是基本 token ， which is not an elementary token,
计算相关损失。 calculate the associated loss.
然后，比较它们之间的这些损失。 Then, compare these losses between them.
我们将删除的 token The token which we will remove
是对损失影响最小的 token ， is the token which impacts the least the loss,
这里是 token “bu”。 here the token 'bu'.
我们在视频开头提到过 We had mentioned at the beginning of the video
在每次迭代中我们可以删除 that at each iteration we could remove
迭代中标记的 p 百分比。 p-percent of the tokens by iteration.
可以在本次迭代中删除的第二个 token The second token that could be removed at this iteration
是 token “du”。 is the token 'du'.
就是这样。 And that's it.
我们只需要重复这些步骤 We just have to repeat these steps
直到我们得到所需大小的词汇表。 until we get the vocabulary of the desired size.
最后一件事。 One last thing.
在实践中，当我们用 Unigram 模型标记一个词时， In practice, when we tokenize a word with a Unigram model,
我们不计算概率的集合 we don't compute the set of probabilities of
一个词所有可能的拆分 all the possible splits of a word
在比较它们以保留最好的之前 before comparing them to keep the best one
但是我们使用 Viterbi 算法 but we use the Viterbi algorithm
这是更有效的方法。 which is much more efficient way to do it.
就是这样！ And that's it!
我希望这个例子 I hope that this example
已经让你更好的了解 has allowed you to better understand
Unigram 分词算法。 the Unigram tokenization algorithm.
（空气呼啸） (air whooshing)