(graphics whooshing)
- How to pre-process pairs of sentences.
We have seen how to tokenize single sentences
and batch them together in the,
"Batching inputs together video."
If this code look unfamiliar to you,
be sure to check that video again.
Here will focus on tasks that classify pair of sentences.
For instance, we may want to classify whether two texts
are paraphrased or not.
Here is an example taken from the Quora Question Pairs
dataset, which focuses on identifying duplicate questions.
In the first pair, the two questions are duplicates,
in the second they are not.
Another pair classification problem is
when we want to know if two sentences are
logically related or not,
a problem called natural language inference or NLI.
In this example, taken from the MultiNLI data set,
we have a pair of sentences for each possible label.
Contradiction, natural or entailment,
which is a fancy way of saying the first sentence
implies the second.
So classifying pairs of sentences is a problem
worth studying.
In fact, in the GLUE benchmark,
which is an academic benchmark for text classification
eight of the 10 data sets are focused
on tasks using pairs of sentences.
That's why models like BERT
are often pre-trained with a dual objective.
On top of the language modeling objective,
they often have an objective related to sentence pairs.
For instance, during pretraining BERT is shown
pairs of sentences and must predict both
the value of randomly masked tokens, and whether the second
sentence follow from the first or not.
Fortunately, the tokenizer from the Transformers library
has a nice API to deal with pairs of sentences.
You just have to pass them as two arguments
to the tokenizer.
On top of the input IDs and the attention mask
we studied already,
it returns a new field called token type IDs,
which tells the model which tokens belong
to the first sentence,
and which ones belong to the second sentence.
Zooming in a little bit, here has an input IDs
aligned with the tokens they correspond to,
their respective token type ID and attention mask.
We can see the tokenizer also added special tokens.
So we have a CLS token, the tokens from the first sentence,
a SEP token, the tokens from the second sentence,
and a final SEP token.
If we have several pairs of sentences,
we can tokenize them together by passing the list
of first sentences, then the list of second sentences
and all the keyword arguments we studied already
like padding=True.
Zooming in at the result,
we can see also tokenize added padding
to the second pair sentences to make the two outputs
the same length, and properly dealt with token type IDs
and attention masks for the two sentences.
This is then all ready to pass through our model.