Welcome to the Hugging Face tasks series!
In this video we’ll take a look at Causal Language Modeling.
Causal language modeling is the task of predicting the next
word in a sentence, given all the previous words. This task is very
similar to the autocorrect function that you might have on your phone.
These models take a sequence to be completed and outputs the complete sequence.
Classification metrics can’t be used as there’s no single correct answer for completion.
Instead, we evaluate the distribution of the text completed by the model.
A common metric to do so is the cross-entropy loss. Perplexity is
also a widely used metric and it is calculated as the exponential of the cross-entropy loss.
You can use any dataset with plain text and tokenize the text to prepare the data.
Causal language models can be used to generate code.
For more information about the Causal Language Modeling task, check out the Hugging Face course.