（屏幕呼啸） (screen whooshing)
（贴纸弹出） (sticker popping)
（屏幕呼啸） (screen whooshing)
- 什么是 BLEU 指标？ - What is the BLEU metric?
对于许多 NLP 任务，我们可以使用常见指标 For many NLP tasks we can use common metrics
比如准确性或 F1 分数， like accuracy or F1 score, but what do you do
但是当你想衡量模型所翻译的文本的质量时 when you wanna measure the quality of text
该如何评估呢？ that's been translated from a model?
在本视频中，我们将为大家介绍一个 In this video, we'll take a look at a widely used metric
广泛使用于机器翻译的指标，叫做 BLEU 。 for machine translation called BLEU.
BLEU 背后的基本逻辑是 The basic idea behind BLEU is to assign
为每个翻译分配一个单独的数字评分 a single numerical score to a translation
用于评估 that tells us how good it is
它与一个或者多个翻译相比，其质量的优劣。 compared to one or more reference translations.
在这个例子中，我们有一个西班牙语句子 In this example, we have a sentence in Spanish
已通过某种模型翻译成英文。 that has been translated into English by some model.
如果我们将生成的翻译 If we compare the generated translation
与一些用于参照的人工翻译进行比较， to some reference human translations,
我们可以看到这个模型其实还不错， we can see that the model is actually pretty good,
但犯了一个常见的错误。 but has made a common error.
西班牙语单词 tengo 在英语中的意思是 have， The Spanish word tengo means have in English,
这种一一对应的直译不太自然。 and this one-to-one translation is not quite natural.
那么对于使用某种自动的方法生成的翻译 So how can we measure the quality
我们如何来评估它的质量呢？ of a generated translation in some automatic way?
BLEU 采用的方法是 The approach that BLEU takes is to compare the n-grams
将所生成翻译的 n-gram 和参照的翻译的 n-gram of the generated translation to the n-grams
进行比较。 in the references.
现在，n-gram 只是用于描述 n 个单词的 Now, an n-gram is just a fancy way of saying
一种奇特的说法。 a chunk of n words.
所以让我们从 unigrams 开始， So let's start with unigrams,
它对应于句子中的单个单词。 which corresponds to the individual words in a sentence.
在此示例中，你可以看到所生成的翻译中有四个单词 In this example, you can see that four of the words
在参照的翻译的其中一个 in the generated translation are also found
也出现了。 in one of the reference translations.
一旦我们找到了匹配项， And once we've found our matches,
给译文打分的一种方法 one way to assign a score to the translation
是计算 unigrams 的精度。 is to compute the precision of the unigrams.
这意味着我们在生成的和参考的翻译中 This means we just count the number of matching words
只计算匹配词的数量 in the generated and reference translations
并且通过除以生成结果的单词数 and normalize the count by dividing by the number of words
来归一化计数值。 in the generation.
在这个例子中，我们找到了四个匹配的词 In this example, we found four matching words
而我们的生成结果中有五个单词。 and our generation has five words.
现在，一般来说，精度范围从零到一， Now, in general, precision ranges from zero to one,
更高的精度分数意味着更好的翻译。 and higher precision scores mean a better translation.
但是到这里还没有结束 But this isn't really the whole story
因为 unigram 精度有一个问题 because one problem with unigram precision
翻译模型有时会在重复的句式上卡住 is that translation models sometimes get stuck
这样的句式会很多次重复 in repetitive patterns and just repeat the same word
某一个单词。 several times.
如果我们只计算单词匹配的数量， If we just count the number of word matches,
我们可以获得非常高的精度分数 we can get really high precision scores
即使从人类的角度来看 even though the translation is terrible
这个翻译很糟糕！ from a human perspective!
例如，如果我们的模型只生成单词 six， For example, if our model just generates the word six,
我们得到了完美的 unigram 精度分数。 we get a perfect unigram precision score.
所以为了解决这个问题，BLEU 使用了修改后的精度 So to handle this, BLEU uses a modified precision
基于它出现在参考翻译中 that clips the number of times to count a word,
出现的最大次数 based on the maximum number of times
再减掉计算一个单词的次数。 it appears in the reference translation.
在这个例子中，单词 six 只在参考翻译中出现了一次 In this example, the word six only appears once
所以我们把分子改为 1 in the reference, so we clip the numerator to one
和修改后的 unigram 精度 and the modified unigram precision
现在给出的分数比预期的要低得多。 now gives a much lower score as expected.
unigram 精度的另一个问题 Another problem with unigram precision
是它没有考虑到 is that it doesn't take into account
单词在翻译中出现的顺序。 the order in which the words appear in the translations.
例如，假设我们有 Yoda 为我们 For example, suppose we had Yoda
翻译西班牙语句子， translate our Spanish sentence,
那么我们可能会得到一些退步的结果， then we might get something backwards like,
比如，“Years sixty thirty have I.” "Years sixty thirty have I."
在这种情况下，修改后的 unigram 精度值 In this case, the modified unigram precision
给出了高精度，这并不是我们真正想要的。 gives a high precision which is not really what we want.
所以要处理词序问题， So to deal with word ordering problems,
BLEU 实际上计算几个不同的 n-gram 精度值， BLEU actually computes the precision
然后对结果计算平均值。 for several different n-grams and then averages the result.
例如，如果我们比较 4-grams， For example, if we compare 4-grams,
我们可以看到翻译中 we can see that there are no matching chunks
没有匹配四个词的语块， of four words in the translations,
所以 4-gram 精度为 0。 and so the 4-gram precision is 0.
现在，使用 Datasets 库计算 BLEU 分数 Now, to compute BLEU scores in Datasets library
真的很简单。 is really very simple.
你只需使用 load_metric 函数， You just use the load_metric function,
为模型的预测提供参考 provide your model's predictions with their references
然后就一切就绪！ and you're good to go!
输出将包含几个重点的字段。 The output will contain several fields of interest.
精度字段包含 The precisions field contains
每个 n-gram 的全部单体精度分数。 all the individual precision scores for each n-gram.
然后 BLEU 分数本身 The BLEU score itself is then calculated
通过取精度分数的几何平均值进行计算。 by taking the geometric mean of the precision scores.
默认情况下，所有四个 n-gram 精度的平均值都会输出， And by default, the mean of all four n-gram precisions
该指标有时也称为 BLEU-4。 is reported, a metric that is sometimes also called BLEU-4.
在此示例中，我们可以看到 BLEU 分数为零 In this example, we can see the BLEU score is zero
因为 4-gram 精度为零。 because the 4-gram precision was zero.
现在，BLEU 指标有一些不错的特性， Now, the BLEU metric has some nice properties,
但这离作为完美评估指标的距离还很远。 but it is far from a perfect metric.
好的特性是它很容易计算 The good properties are that it's easy to compute
它被广泛用于研究 and it's widely used in research
这样你就可以基于普遍的基准将你的模型 so you can compare your model against others
与其他模型进行比较。 on common benchmarks.
另一方面，BLEU 有几个大问题， On the other hand, there are several big problems with BLEU,
包括实际上它不包含语义 including the fact it doesn't incorporate semantics
它不适用于非英语语言。 and it struggles a lot on non-English languages.
BLEU 的另一个问题 Another problem with BLEU
是它假定人工翻译 is that it assumes the human translations
已经被词元化 have already been tokenized
这使得在基于不同的分词器的情况下 and this makes it hard to compare models
比较模型变得困难。 that use different tokenizers.
所以正如我们所见，衡量文本的质量 So as we've seen, measuring the quality of texts
仍然是 NLP 研究中的一个困难和开放的问题。 is still a difficult and open problem in NLP research.
对于机器翻译，目前的推荐 For machine translation, the current recommendation
是使用 SacreBLEU 指标， is to use the SacreBLEU metric,
它解决了 BLEU 的词元化限制。 which addresses the tokenization limitations of BLEU.
正如你在此示例中所见， As you can see in this example,
计算 SacreBLEU 分数几乎与 BLEU computing the SacreBLEU score is almost identical
完全一致。 to the BLEU one.
主要区别在于我们现在传递一个文本列表 The main difference is that we now pass a list of texts
而不是翻译的单词列表， instead of a list of words to the translations,
SacreBLEU 负责底层的词元化。 and SacreBLEU takes care of the tokenization under the hood.
（屏幕呼啸） (screen whooshing)