(subtle blast)
- How to batch inputs together.
In this video, we will see how
to batch input sequences together.
In general, the sentences we want to pass through our model
won't all have the same lengths.
Here, we are using the model we saw
in the sentiment analysis pipeline
and want to classify two sentences.
When tokenizing them and mapping each token
to its corresponding input IDs,
we get two lists of different lengths.
Trying to create a tensor or a NumPy array
from those two lists will result in an error,
because all arrays and tensors should be rectangular.
One way to overcome this limit
is to make the second sentence
the same length as the first
by adding a special token as many times as necessary.
Another way would be to truncate the first sequence
to the length of the second,
but we would them lose a lot of information
that might be necessary to properly classify the sentence.
In general, we only truncate sentences
when they are longer than the maximum length
the model can handle.
The value used to pad the second sentence
should not be picked randomly;
the model has been pretrained with a certain padding ID,
which you can find in tokenizer.pad_token_id.
Now that we have padded our sentences,
we can make a batch with them.
If we pass the two sentences to the model separately
and batched together however,
we notice that we don't get the same results
for the sentence that is padded,
here, the second one.
It's at the back in the Transformers Library? No.
If you remember that Transformer models
make heavy use of attention layers,
this should not come as a total surprise;
when computing the contextual representation of each token,
the attention layers look at all the other words
in the sentence.
If we have just the sentence
or the sentence with several padding tokens added,
it's logical we don't get the same values.
To get the same results with or without padding,
we need to indicate to the attention layers
that they should ignore those padding tokens.
This is done by creating an attention mask,
a tensor with the same shape as the input IDs,
with zeros and ones.
Ones indicate the tokens the attention layers
should consider in the context
and zeros the tokens they should ignore.
Now, passing this attention mask along with the input ID
will give us the same results
as when we sent the two sentences individually to the model.
This is all done behind the scenes by the tokenizer
when you apply it to several sentences
with the flag padding=True.
It will apply the padding with the proper value
to the smaller sentences
and create the appropriate attention mask.
(subtle blast)