(whooshing)
- Let's see how we can preprocess our data
for masked language modeling.
As a reminder, masked language modeling
is when a model needs to fill the blanks in a sentence.
To do this, you just need texts, no labels,
as this is a self-supervised problem.
To apply this on your own data,
just make sure you have all your texts gathered
in one column of your dataset.
Before we start randomly masking things,
we will need to somehow make all those texts the same length
to batch them together.
The first way to make all the texts the same length
is the one we used in text classification.
Let's pad the short texts and truncate the long ones.
As we have seen
when we processed data for text classification,
this is all done by our tokenizer
with the right options for padding and truncation.
This will however make us lose a lot of texts
if the examples in our dataset are very long,
compared to the context length we picked.
Here, all the portion in gray is lost.
This is why a second way to generate samples of text
with the same length is to chunk our text
in pieces of context lengths,
instead of discarding everything after the first chunk.
There will probably be a remainder
of length smaller than the context size,
which we can choose to keep and pad or ignore.
Here is how we can apply this in practice,
by just adding the return overflowing tokens option
in our tokenizer call.
Note how this gives us a bigger dataset!
This second way of chunking is ideal if all your texts
are very long, but it won't work
as nicely if you have a variety of lengths in the texts.
In this case,
the best option is to concatenate all your tokenized texts
in one big stream, with a special tokens
to indicate when you pass from one document to the other,
and only then split the big stream into chunks.
Here is how it can be done with code,
with one loop to concatenate all the texts
and another one to chunk it.
Notice how it reduces the number of samples
in our dataset here,
there must have been quite a few short entries!
Once this is done, the masking is the easy part.
There is a data collator designed specifically for this
in the Transformers library.
You can use it directly in the Trainer,
or when converting your datasets to tensorflow datasets
before doing Keras.fit, with the to_tf_dataset method.
(whooshing)