(air whooshing)
- How to preprocess pairs of sentences?
We have seen how to tokenize single sentences
and batch them together
in the "Batching inputs together" video.
If this code looks unfamiliar to you,
be sure to check that video again!
Here, we will focus on tasks
that classify pairs of sentences.
For instance, we may want to classify
whether two texts are paraphrases or not.
Here is an example taken
from the Quora Question Pairs dataset,
which focuses on identifying duplicate questions.
In the first pair, the two questions are duplicates;
in the second, they are not.
Another classification problem
is when we want to know if two sentences
are logically related or not,
a problem called Natural Language Inference or NLI.
In this example taken from the MultiNLI dataset,
we have a pair of sentences for each possible label:
contradiction, neutral or entailment,
which is a fancy way of saying the first sentence
implies the second.
So classifying pairs of sentences
is a problem worth studying.
In fact, in the GLUE benchmark,
which is an academic benchmark for text classification,
eight of the 10 datasets are focused on tasks
using pairs of sentences.
That's why models like BERT are often pretrained
with a dual objective:
on top of the language modeling objective,
they often have an objective related to sentence pairs.
For instance, during pretraining,
BERT is shown pairs of sentences
and must predict both the value of randomly masked tokens
and whether the second sentence
follows from the first or not.
Fortunately, the tokenizer from the Transformers library
has a nice API to deal with pairs of sentences:
you just have to pass them as two arguments
to the tokenizer.
On top of the input IDs
and the attention mask we studied already,
it returns a new field called token type IDs,
which tells the model which tokens
belong to the first sentence
and which ones belong to the second sentence.
Zooming in a little bit, here are the input IDs,
aligned with the tokens they correspond to,
their respective token type ID and attention mask.
We can see the tokenizer also added special tokens
so we have a CLS token, the tokens from the first sentence,
a SEP token, the tokens from the second sentence,
and a final SEP token.
If we have several pairs of sentences,
we can tokenize them together
by passing the list of first sentences,
then the list of second sentences
and all the keyword arguments we studied already,
like padding=True.
Zooming in at the result,
we can see how the tokenizer added padding
to the second pair of sentences,
to make the two outputs the same length.
It also properly dealt with token type IDS
and attention masks for the two sentences.
This is then all ready to pass through our model!
(air whooshing)