Welcome to the Hugging Face tasks series.
In this video, we will take a look at the Translation task.
Translation is the task of translating text from one language to another.
These models take a text in the source language and output the translation of that text in
the target language.
The task is evaluated on the BLEU score.
The score ranges from 0 to 1, in which 1 means the translation perfectly matched and 0 did
not match at all.
BLEU is calculated over subsequent tokens called n-grams.
Unigram refers to a single token while bi-gram refers to token pairs and n-grams refer to
n subsequent tokens.
Machine translation datasets contain pairs of text in a language and translation of the
text in another language.
These models can help you build conversational agents across different languages.
One option is to translate the training data used for the chatbot and train a separate
chatbot.
You can put one translation model from your user’s language to the language your chatbot
is trained on, translate the user inputs and do intent classification, take the output
of the chatbot and translate it from the language your chatbot was trained on to the user’s
language.
For more information about the Translation task, check out the Hugging Face course.