(logo whooshing)
- What is the ROUGE metric?
For many NLP tasks
we can use common metrics like accuracy or F1 score.
But what do you do when you wanna measure something
like the quality of a summary from a model like T5?
In this video, we'll take a look
at a widely used metric for text summarization called ROUGE.
There are actually several variants of ROUGE
but the basic idea behind all of them
is to assign a single numerical score
to a summary that tells us how good it is
compared to one or more reference summaries.
In this example, we have a book review
that has been summarized by some model.
If we compare the generated summary
to some reference human summaries, we can see
that the model is actually pretty good
and only differs by a word or two.
So how can we measure the quality
of a generated summary in an automatic way?
The approach that ROUGE takes is to compare the n-grams
of the generated summary to the n-grams of the references.
And n-gram is just a fancy way of saying a chunk of N words.
So let's start with unigrams
which correspond to the individual words in a sentence.
In this example, you can see that six
of the words in the generated summary are also found
in one of the reference summaries.
And the rouge metric that compares unigrams
is called ROUGE-1.
Now that we found our matches, one way to assign a score
to the summary is to compute the recall of the unigrams.
This means we just count the number
of matching words in the generated
and reference summaries and normalize the count
by dividing by the number of words in the reference.
In this example, we found six matching words
and our reference has six words.
So our unigram recall is perfect.
This means that all of the words
in the reference summary have been produced
in the generated one.
Now, perfect recall sounds great, but imagine
if our generated summary have been something like
I really, really, really,
really loved reading the Hunger Games.
This would also have perfect recall
but is arguably a worse summary,
since it is verbose.
To deal with these scenarios,
we can also compute precision,
which in the ROUGE context measures how much
of the generator summary was relevant.
In practice, both precision and recall are usually computed
and then the F1 score is reported.
Now we can change the granularity
of the comparison by comparing bigrams instead of unigrams.
With bigrams, we chunk the sentence into pairs
of consecutive words and then count how many pairs
in the generated summary are present in the reference one.
This gives us ROUGE-2 precision and recall
which as we can see,
is lower than the ROUGE-1 scores from earlier.
Now, if the summaries are long,
the ROUGE-2 scores will generally be small
because there are fewer bios to match.
And this is also true for abstractive summarization.
So both ROUGE-1 and ROUGE-2 scores are usually reported.
The last ROUGE variant we will discuss is ROUGE L.
ROUGE L doesn't compare ngrams
but instead treats each summary as a sequence of words
and then looks for the longest common subsequence or LCS.
A subsequence is a sequence that appears
in the same relative order, but not necessarily contiguous.
So in this example, I loved reading the Hunger Games,
is the longest common subsequence between the two summaries.
And the main advantage of ROUGE L
over ROUGE-1 or ROUGE-2 is that it doesn't depend
on consecutive n-gram matches, and so it tends
to capture sentence structure much more accurately.
Now to compute ROUGE scores
in the data sets library is very simple.
You just use the load_metric function,
provide your model summaries along with the references
and you're good to go.
The output from the calculation
contains a lot of information.
The first thing we can see is that the confidence intervals
of each ROUGE score are provided
in the low, mid and high fields.
This is really useful if you wanna know the spread
of your ROUGE scores when comparing two or more models.
The second thing to notice
is that we have four types of ROUGE score.
We've already seen ROUGE-1, ROUGE-2 and ROUGE-L
So what is ROUGE-L sum?
Well, the sum in ROUGEL's sum
refers to the fact that this metric is computed
over a whole summary while ROUGE-L is computed
as the average of individual sentences.
(logo whooshing)