(logo whooshes)
- How to batch inputs together.
In this video, we'll see
how to batch input sequences together.
In general, the sentences we want to pass
through our model won't all have the same lengths.
Here, we are using the model we saw
in the sentiment analysis pipeline
and want to classify two sentences.
When tokenizing them and mapping each token
to its corresponding input IDs,
we get two lists of different lengths.
Trying to create a tensor and NumPy array
from those two lists will result in an error
because all arrays and tensors should be rectangular.
One way to overcome this limit
is to make the second sentence the same length as the first
by adding a special token as many times as necessary.
Another way would be to truncate the first sequence
to the length of the second, but we would then lose a lot
of information that may be necessary
to properly classify the sentence.
In general, we only truncate sentences when they are longer
than the maximum length the model can handle.
The value used to pad the second sentence
should not be picked randomly.
The model has been pretrained with a certain padding ID,
which you can find in tokenizer.pad_token_id.
Now that we have padded our sentences,
we can make a batch with them.
If we pass the two sentences to the model separately
or batched together, however, we notice
that we don't get the same results
for the sentence that is padded.
Here, the second one.
Expect the word in the transformer library?
No.
If you remember that Transformer models make heavy use
of attention layers, it should not come as a total surprise.
When computing the contextual representation of each token,
the attention layers look at all the other words
in the sentence.
If we have just a sentence
or the sentence with several padding tokens added,
it's logical we don't get the same values.
To get the same results with or without padding,
we need to indicate to the attention layers
that they should ignore those padding tokens.
This is done by creating an attention mask,
a tensor with the same shape as the input IDs
with zeros and ones.
Ones indicate the tokens the attention layers
should consider in the context,
and zeros, the tokens they should ignore.
Now, passing this attention mask
along with the input IDs will give us the same results
as when we sent the two sentences individually to the model.
This is all done behind the scenes by the tokenizer
when you apply it to several sentences
with the flag padding equals true.
It will apply the padding
with the proper value to the smaller sentences
and create the appropriate attention mask.