(air whooshing)
(smiley clicking)
(air whooshing)
- Let's see how to preprocess a dataset for summarization.
This is the task of, well, summarizing a long document.
This video will focus on how to preprocess your dataset
once you have managed to put it in the following format:
one column for the long documents,
and one for the summaries.
Here is how we can achieve this
with the Datasets library on the XSUM dataset.
As long as you manage to have your data look like this,
you should be able to follow the same steps.
For once, our labels are not integers
corresponding to some classes, but plain text.
We will thus need to tokenize them, like our inputs.
There is a small trap there though,
as we need to tokenize our targets
inside the as_target_tokenizer context manager.
This is because the special tokens we add
might be slightly different for the inputs and the target,
so the tokenizer has to know which one it is processing.
Processing the whole dataset is then super easy
with the map function.
Since the summaries are usually much shorter
than the documents, you should definitely pick
different maximum lengths for the inputs and targets.
You can choose to pad at this stage to that maximum length
by setting padding=max_length.
Here we'll show you how to pad dynamically,
as it requires one more step.
Your inputs and targets are all sentences
of various lengths.
We'll pad the inputs and targets separately
as the maximum lengths of the inputs and targets
are completely different.
Then, we pad the inputs to the maximum lengths
among the inputs, and same for the target.
We pad the input with the pad token,
and the targets with the -100 index
to make sure they are not taken into account
in the loss computation.
The Transformers library provide us
with a data collator to do this all automatically.
You can then pass it to the Trainer with your datasets,
or use it in the to_tf_dataset method before using model.fit
on your current model.
(air whooshing)