(air whooshing)
- What is dynamic padding?
In the "Batching Inputs together" video,
we have seen that to be able to group inputs
of different lengths in the same batch,
we need to add padding tokens to all the short inputs
until they are all of the same length.
Here, for instance, the longest sentence is the third one,
and we need to add five, two, or seven pad tokens
to the other sentences to have four sentences
of the same lengths.
When dealing with a whole dataset,
there are various padding strategies we can apply.
The most obvious one is to pad all the elements
of the dataset to the same length:
the length of the longest sample.
This will then give us batches
that all have the same shape
determined by the maximum sequence length.
The downside is that batches composed from short sentences
will have a lot of padding tokens
which will introduce more computations in the model
we ultimately don't need.
To avoid this, another strategy is to pad the elements
when we batch them together,
to the longest sentence inside the batch.
This way, batches composed of short inputs will be smaller
than the batch containing the longest sentence
in the dataset.
This will yield some nice speedup on CPU and GPU.
The downside is that all batches
will then have different shapes,
which slows down training on accelerators like TPUs.
Let's see how to apply both strategies in practice.
We have actually seen how to apply fixed padding
in the Datasets Overview video,
when we preprocessed the MRPC dataset:
after loading the dataset and tokenizer,
we applied the tokenization to all the dataset
with padding and truncation
to make all samples of length 128.
As a result, if we pass this dataset
to a PyTorch DataLoader,
we get batches of shape batch size, here 16, by 128.
To apply dynamic padding,
we must defer the padding to the batch preparation,
so we remove that part from our tokenize function.
We still leave the truncation part
so that inputs that are bigger than the maximum length
accepted by the model, usually 512,
get truncated to that length.
Then we pad our samples dynamically
by using a data collator.
Those classes in the Transformers library
are responsible for applying all the final processing
needed before forming a batch,
here DataCollatorWithPadding will pad the samples
to the maximum length inside the batch of sentences.
We pass it to the PyTorch DataLoader as a collate function,
then observe that the batches generated
have various lengths,
all way below the 128 from before.
Dynamic batching will almost always be faster
on CPUs and GPUs, so you should apply it if you can.
Remember to switch back to fixed padding, however,
if you run your training script on TPU
or need batches of fixed shapes.
(air whooshing)