(screen whooshing)
(sticker popping)
(screen whooshing)
- What is the BLEU metric?
For many NLP tasks we can use common metrics
like accuracy or F1 score, but what do you do
when you wanna measure the quality of text
that's been translated from a model?
In this video, we'll take a look at a widely used metric
for machine translation called BLEU.
The basic idea behind BLEU is to assign
a single numerical score to a translation
that tells us how good it is
compared to one or more reference translations.
In this example, we have a sentence in Spanish
that has been translated into English by some model.
If we compare the generated translation
to some reference human translations,
we can see that the model is actually pretty good,
but has made a common error.
The Spanish word tengo means have in English,
and this one-to-one translation is not quite natural.
So how can we measure the quality
of a generated translation in some automatic way?
The approach that BLEU takes is to compare the n-grams
of the generated translation to the n-grams
in the references.
Now, an n-gram is just a fancy way of saying
a chunk of n words.
So let's start with unigrams,
which corresponds to the individual words in a sentence.
In this example, you can see that four of the words
in the generated translation are also found
in one of the reference translations.
And once we've found our matches,
one way to assign a score to the translation
is to compute the precision of the unigrams.
This means we just count the number of matching words
in the generated and reference translations
and normalize the count by dividing by the number of words
in the generation.
In this example, we found four matching words
and our generation has five words.
Now, in general, precision ranges from zero to one,
and higher precision scores mean a better translation.
But this isn't really the whole story
because one problem with unigram precision
is that translation models sometimes get stuck
in repetitive patterns and just repeat the same word
several times.
If we just count the number of word matches,
we can get really high precision scores
even though the translation is terrible
from a human perspective!
For example, if our model just generates the word six,
we get a perfect unigram precision score.
So to handle this, BLEU uses a modified precision
that clips the number of times to count a word,
based on the maximum number of times
it appears in the reference translation.
In this example, the word six only appears once
in the reference, so we clip the numerator to one
and the modified unigram precision
now gives a much lower score as expected.
Another problem with unigram precision
is that it doesn't take into account
the order in which the words appear in the translations.
For example, suppose we had Yoda
translate our Spanish sentence,
then we might get something backwards like,
"Years sixty thirty have I."
In this case, the modified unigram precision
gives a high precision which is not really what we want.
So to deal with word ordering problems,
BLEU actually computes the precision
for several different n-grams and then averages the result.
For example, if we compare 4-grams,
we can see that there are no matching chunks
of four words in the translations,
and so the 4-gram precision is 0.
Now, to compute BLEU scores in Datasets library
is really very simple.
You just use the load_metric function,
provide your model's predictions with their references
and you're good to go!
The output will contain several fields of interest.
The precisions field contains
all the individual precision scores for each n-gram.
The BLEU score itself is then calculated
by taking the geometric mean of the precision scores.
And by default, the mean of all four n-gram precisions
is reported, a metric that is sometimes also called BLEU-4.
In this example, we can see the BLEU score is zero
because the 4-gram precision was zero.
Now, the BLEU metric has some nice properties,
but it is far from a perfect metric.
The good properties are that it's easy to compute
and it's widely used in research
so you can compare your model against others
on common benchmarks.
On the other hand, there are several big problems with BLEU,
including the fact it doesn't incorporate semantics
and it struggles a lot on non-English languages.
Another problem with BLEU
is that it assumes the human translations
have already been tokenized
and this makes it hard to compare models
that use different tokenizers.
So as we've seen, measuring the quality of texts
is still a difficult and open problem in NLP research.
For machine translation, the current recommendation
is to use the SacreBLEU metric,
which addresses the tokenization limitations of BLEU.
As you can see in this example,
computing the SacreBLEU score is almost identical
to the BLEU one.
The main difference is that we now pass a list of texts
instead of a list of words to the translations,
and SacreBLEU takes care of the tokenization under the hood.
(screen whooshing)