Welcome to the Hugging Face tasks series!
In this video we’ll take a look at Masked Language Modeling.
Masked language modeling is the task of predicting which words should fill in the blanks of a
sentence.
These models take a masked text as the input and output the possible values for that mask.
Masked language modeling is handy before fine-tuning your model for your task.
For example, if you need to use a model in a specific domain, say, biomedical documents,
models like BERT will treat your domain-specific words as rare tokens.
If you train a masked language model using your biomedical corpus and then fine tune
your model on a downstream task, you will have a better performance.
Classification metrics can’t be used as there’s no single correct answer to mask
values.
Instead, we evaluate the distribution of the mask values.
A common metric to do so is the cross-entropy loss.
Perplexity is also a widely used metric and it is calculated as the exponential of the
cross-entropy loss.
You can use any dataset with plain text and tokenize the text to mask the data.
For more information about the Masked Language Modeling, check out the Hugging Face course.