(whooshing sound)
(sticker popping)
(whooshing sound)
In a lot of our examples,
you're going to see DataCollators popping up over and over.
They're used in both PyTorch and TensorFlow workflows,
and maybe even in JAX,
but no-one really knows what's happening in JAX.
We do have a research team working on it though,
so maybe they'll tell us soon.
But coming back on topic.
What are data collators?
Data collators collate data.
That's not that helpful.
But to be more specific, they put together a list of samples
into a single training minibatch.
For some tasks,
the data collator can be very straightforward.
For example, when you're doing sequence classification,
all you really need from your data collator
is that it pads your samples to the same length
and concatenates them into a single Tensor.
But for other workflows, data collators can be quite complex
as they handle some of the preprocessing
needed for that particular task.
So, if you want to use a data collator,
for PyTorch users, you usually pass the data collator
to your Trainer object.
In TensorFlow, it's a bit different.
The easiest way to use a data collator
is to pass it to the to_tf_dataset method of your dataset.
And this will give you a tensorflow_tf_data.dataset
that you can then pass to model.fit.
You'll see these approaches used in the examples
and notebooks throughout this course.
Also note that all of our collators
take a return_tensors argument.
You can set this to "pt" to get PyTorch Tensors,
"tf" to get TensorFlow Tensors,
or "np" to get Numpy arrays.
For backward compatibility reasons,
the default value is "pt",
so PyTorch users don't even have to set this argument
most of the time.
And so as a result, they're often totally unaware
that this argument even exists.
We can learn something from this
which is that the beneficiaries of privilege
are often the most blind to its existence.
But okay, coming back.
Let's see how some specific data collators work in action.
Although again, remember if none
of the built-in data collators do what you need,
you can always write your own and they're often quite short.
So first, we'll see the "basic" data collators.
These are DefaultDataCollator and DataCollatorWithPadding.
These are the ones you should use
if your labels are straightforward
and your data doesn't need any special processing
before being ready for training.
Notice that because different models
have different padding tokens,
DataCollatorWithPadding will need your model's Tokenizer
so it knows how to pad sequences properly.
The default data collator doesn't need a Tokenizer to work,
but it will as a result throw an error
unless all of your sequences are the same length.
So, you should be aware of that.
Moving on though.
A lot of the other data collators
aside from the basic two are,
they're usually designed to handle one specific task.
And so, I'm going to show a couple here.
These are DataCollatorForTokenClassification
and DataCollatorForSeqToSeq.
And the reason these tasks need special collators
is because their labels are variable in length.
In token classification there's one label for each token,
and so the length of the labels
is the length of the sequence.
While in SeqToSeq the labels are a sequence of tokens
that can be variable length,
that can be very different
from the length of the input sequence.
So in both of these cases, we handle collating that batch
by padding the labels as well,
as you can see here in this example.
So, inputs and the labels will need to be padded
if we want to join samples of variable length
into the same minibatch.
That's exactly what the data collators
and that's exactly what these data collators will do for us
you know, for this particular task.
So, there's one final data collator
I want to show you as well just in this lecture.
And that's the DataCollatorForLanguageModeling.
So, it's very important, and it's firstly,
because language models are just so foundational
to do for everything we do with NLP these days.
But secondly, because it has two modes
that do two very different things.
So you choose which mode you want with the mlm argument.
Set it to True for masked language modeling,
and set it to False for causal language modeling.
So, collating data for causal language modeling
is actually quite straightforward.
The model is just making predictions
for what token comes next, and so your labels
are more or less just a copy of your inputs,
and the collator will handle that
and ensure that the inputs and labels are padded correctly.
When you set mlm to True though,
you get quite different behavior,
that's different from any other data collator,
and that's because setting mlm to True
means masked language modeling
and that means the labels need to be,
you know, the inputs need to be masked.
So, what does that look like?
So, recall that in masked language modeling,
the model is not predicting the next word,
instead we randomly mask out some tokens
and the model predicts all of them at once.
So, it tries to kinda fill in the blanks
for those masked tokens.
But the process of random masking is surprisingly complex.
If we follow the protocol from the original BERT paper,
we need to replace some tokens with a masked token,
some other tokens with a random token,
and then keep a third set of tokens unchanged.
Yeah, this is not the lecture
to go into the specifics of that or why we do it.
You can always check out the original BERT paper
if you're curious.
It's well written. It's easy to understand.
The main thing to know here is that it can be a real pain
and quite complex to implement that yourself.
But DataCollatorForLanguageModeling will do it for you
when you set mlm to True.
And that's an example of the more intricate
preprocessing that some of our data collators do.
And that's it!
So, this covers the most commonly used data collators
and the tasks they're used for.
And hopefully, now you'll know when to use data collators
and which one to choose for your specific task.
(whooshing sound)