(screen whooshing)
(sticker popping)
(screen whooshing)
- In this video, we take a look
at the mysterious sounding metric called perplexity.
You might have encountered perplexity
when reading about generative models.
You can see two examples here,
one from the original transformer paper,
"Attention is all you need,"
and the other one from the more recent GPT-2 paper.
Perplexity is a common metric to measure the performance
of language models.
The smaller its value, the better the performance.
But what does it actually mean and how can we calculate it?
A very common quantity in machine learning
is the likelihood.
We can calculate the likelihood
as the product of each token's probability.
What this means is that for each token,
we use the language model to predict its probability
based on the previous tokens.
In the end, we multiply all probabilities
to get the likelihood.
With the likelihood,
we can calculate another important quantity,
the cross-entropy.
You might have already heard about cross-entropy
when looking at loss function.
It is often used as a loss function in classification.
In language modeling, we predict the next token
based on the previous token,
which is also a classification task.
Therefore, if we want to calculate
the cross-entropy of an example,
we can simply pass it to the model
with its inputs as labels.
The loss then corresponds to the cross-entropy.
We are now only a single operation away
from calculating the perplexity.
By exponentiating the cross-entropy, we get the perplexity.
So you see that the perplexity is closely related
to the loss.
Plugging in previous results
shows that this is equivalent to exponentiating
the negative average lock probability of each token.
Keep in mind that the loss is only a weak proxy
for a model's ability to generate quality text
and the same is true for perplexity.
For this reason, one usually also calculates
more sophisticated metrics
such as BLEU or ROUGE on generative tasks.
(screen whooshing)