（徽标呼啸而过） (logo whooshing)
- 什么是 ROUGE 指标？ - What is the ROUGE metric?
对于许多 NLP 任务 For many NLP tasks
我们可以使用常见的指标，如准确性或 F1 分数。 we can use common metrics like accuracy or F1 score.
但是当你想评估类似从像 T5 这样的模型上 But what do you do when you wanna measure something
所获得的文本摘要的质量，该如何操作呢？ like the quality of a summary from a model like T5?
在本视频中，我们将了解 In this video, we'll take a look
被广泛用于文本摘要的评估指标，称为 ROUGE。 at a widely used metric for text summarization called ROUGE.
ROUGE 实际上有几种变体 There are actually several variants of ROUGE
但所有这些变体背后的基本思想 but the basic idea behind all of them
是为每个摘要分配一个单独的分数 is to assign a single numerical score
来告诉我们相比一个或者多个参考的摘要 to a summary that tells us how good it is
当前的摘要有多好。 compared to one or more reference summaries.
在这个例子中，我们有一个书评 In this example, we have a book review
它是通过某些模型摘要获得。 that has been summarized by some model.
如果我们将生成的摘要 If we compare the generated summary
和一些人工的摘要相比较， to some reference human summaries,
我们可以看到该模型实际上非常好 we can see that the model is actually pretty good
并且只相差一两个词。 and only differs by a word or two.
那么我们如何通过自动的方式 So how can we measure the quality
评估生成的摘要的质量呢？ of a generated summary in an automatic way?
ROUGE 采用的方法是比较生成的摘要的 n-gram The approach that ROUGE takes is to compare the n-grams
和参考文献的 n-gram。 of the generated summary to the n-grams of the references.
n-gram 只是一种表达 N 个单词的词块的流行说法。 And n-gram is just a fancy way of saying a chunk of N words.
所以让我们从 unigram 开始 So let's start with unigrams
它对应于句子中的各个单词。 which correspond to the individual words in a sentence.
在这个例子中，你可以看到 In this example, you can see that six
在生成的摘要中有 6 个单词，也出现在 of the words in the generated summary are also found
其中一个参考的摘要中。 in one of the reference summaries.
而比较 unigram 的 rouge metric And the rouge metric that compares unigrams
被称为 ROUGE-1。 is called ROUGE-1.
现在我们找到了我们的匹配项，一种为摘要分配分数的方法 Now that we found our matches, one way to assign a score
是计算 unigram 的被召回次数。 to the summary is to compute the recall of the unigrams.
这意味着我们只计算 This means we just count the number
生成摘要和参考摘要的匹配词 of matching words in the generated
并将计数通过除以参考文本中的单词数 and reference summaries and normalize the count
进行规范化处理。 by dividing by the number of words in the reference.
在这个例子中，我们找到了六个匹配的词 In this example, we found six matching words
我们的参考文本有六个词。 and our reference has six words.
所以我们的 unigram 召回是完美的。 So our unigram recall is perfect.
这意味着在参考摘要中 This means that all of the words
的所有的词都会出现 in the reference summary have been produced
在生成的摘要中。 in the generated one.
现在，完美的召回听起来不错，但想象一下 Now, perfect recall sounds great, but imagine
如果我们生成的摘要是这样的 if our generated summary have been something like
我真的，真的，真的， I really, really, really,
真的很喜欢阅读 Hunger Games。 really loved reading the Hunger Games.
这也会有完美的召回 This would also have perfect recall
但可以说是一个更糟糕的总结， but is arguably a worse summary,
因为它很冗长。 since it is verbose.
为了应对这些场景， To deal with these scenarios,
我们还可以计算精度， we can also compute precision,
在 ROUGE 上下文中，它衡量了 which in the ROUGE context measures how much
在生成器摘要中具有多少相关性。 of the generator summary was relevant.
在实际操作中，通常计算精度和召回率 In practice, both precision and recall are usually computed
然后报告 F1 分数。 and then the F1 score is reported.
现在我们可以改变比较的粒度 Now we can change the granularity
将比较的对象由 unigram 改变为 bigram。 of the comparison by comparing bigrams instead of unigrams.
使用 bigram，我们将句子 With bigrams, we chunk the sentence into pairs
切分为成对的连续单词， of consecutive words and then count how many pairs
然后计算生成的摘要中有多少对出现在参考摘要中。 in the generated summary are present in the reference one.
这给了我们 ROUGE-2 精确度和召回率 This gives us ROUGE-2 precision and recall
正如我们所见， which as we can see,
低于之前的 ROUGE-1 分数。 is lower than the ROUGE-1 scores from earlier.
现在，如果摘要很长， Now, if the summaries are long,
ROUGE-2 分数通常会很小 the ROUGE-2 scores will generally be small
因为要匹配的 bios 更少。 because there are fewer bios to match.
这也适用于重写式摘要。 And this is also true for abstractive summarization.
所以通常会报告 ROUGE-1 和 ROUGE-2 分数。 So both ROUGE-1 and ROUGE-2 scores are usually reported.
我们将讨论的最后一个 ROUGE 变体是 ROUGE L。 The last ROUGE variant we will discuss is ROUGE L.
ROUGE L 不比较 ngram ROUGE L doesn't compare ngrams
而是将每个摘要视为一系列单词 but instead treats each summary as a sequence of words
然后寻找最长的公共子序列或 LCS。 and then looks for the longest common subsequence or LCS.
子序列是以相同的相对顺序 A subsequence is a sequence that appears
出现的序列，但不一定是连续的。 in the same relative order, but not necessarily contiguous.
所以在这个例子中，我喜欢阅读 Hunger Games， So in this example, I loved reading the Hunger Games,
是两个摘要之间最长的公共子序列。 is the longest common subsequence between the two summaries.
而相比 ROUGE-1 或 ROUGE-2 And the main advantage of ROUGE L
ROUGE L 的主要优势是它不依赖于 over ROUGE-1 or ROUGE-2 is that it doesn't depend
在连续的 n-gram 匹配上， on consecutive n-gram matches, and so it tends
所以它倾向于更准确地捕捉句子结构。 to capture sentence structure much more accurately.
现在在 Dataset 库中 Now to compute ROUGE scores
计算 ROUGE 分数很简单。 in the data sets library is very simple.
你只需使用 load_metric 函数， You just use the load_metric function,
提供你的模型摘要以及参考摘要 provide your model summaries along with the references
你可以开始了。 and you're good to go.
计算的输出 The output from the calculation
包含很多信息。 contains a lot of information.
我们首先可以看到的是 The first thing we can see is that the confidence intervals
每个 ROUGE 分数的置信区间 of each ROUGE score are provided
每个 ROUGE 分数的置信区间。 in the low, mid and high fields.
当比较两个或多个模型时， This is really useful if you wanna know the spread
如果你想知道 ROUGE 分数的范围，这就真的很有用。 of your ROUGE scores when comparing two or more models.
第二点要注意 The second thing to notice
是我们有四种类型的 ROUGE 分数。 is that we have four types of ROUGE score.
我们已经看过 ROUGE-1、ROUGE-2 和 ROUGE-L We've already seen ROUGE-1, ROUGE-2 and ROUGE-L
那么什么是 ROUGE-L sum 呢？ So what is ROUGE-L sum?
其实就是 ROUGEL 的总和 Well, the sum in ROUGEL's sum
指的是这个指标是 refers to the fact that this metric is computed
当 ROUGE-L 作为单个句子的平均值计算时， over a whole summary while ROUGE-L is computed
基于整个摘要计算出来的。 as the average of individual sentences.
（徽标呼啸而过） (logo whooshing)