(transition music)
- In this video, we take a look at the data processing
necessary to train causal language models.
Causal language modeling is the task
of predicting the next token based on the previous ones.
Another term for causal language modeling
is autoregressive modeling.
In the example that you can see here,
the next token could, for example,
be NLP or it could be machine learning.
A popular example of causal language models
is the GPT family of models.
To train models such as GPT,
we usually start with a large corpus of text files.
These files can be webpages scraped from the internet
such as the Common Crawl dataset
or they can be Python files from GitHub,
like the ones you can see here.
As a first step, we need to tokenize these files
such that we can feed them through the model.
Here, we show the tokenized texts as bars of various length,
illustrating that they're shorter and longer ones.
This is very common when working with text.
However, transform models have a limited context window
and depending on the data source,
it is possible that the tokenized texts
are much longer than this window.
In this case, we could just truncate the sequences
to the context length,
but this would mean that we lose everything
after the first context window.
Using the return overflowing token flag,
we can use the tokenizer to create chunks
with each one being the size of the context length.
Sometimes, it can still happen
that the last chunk is too short
if there aren't enough tokens to fill it.
In this case, we can just remove it.
With the return_length keyword,
we also get the length of each chunk from the tokenizer.
This function shows all the steps
necessary to prepare the dataset.
First, we tokenize the dataset
with the flags I just mentioned.
Then, we go through each chunk
and if it's length matches the context length,
we add it to the inputs we return.
We can apply this function to the whole dataset.
In addition, we make sure
that to use batches and remove the existing columns.
We need to remove the existing columns,
because we can create multiple samples per text,
and the shapes in the dataset
would not match anymore in that case.
If the context length is of similar lengths as the files,
this approach doesn't work so well anymore.
In this example, both sample 1 and 2
are shorter than the context size
and will be discarded with the previous approach.
In this case, it is better to first tokenize each sample
without truncation
and then concatenate the tokenized samples
with an end of string or EOS token in between.
Finally, we can chunk this long sequence
with the context length and we don't lose too many sequences
because they're too short anymore.
So far, we have only talked
about the inputs for causal language modeling,
but not the labels needed for supervised training.
When we do causal language modeling,
we don't require any extra labels for the input sequences
as the input sequences themselves are the labels.
In this example, when we feed the token trans to the model,
the next token we wanted to predict is formers.
In the next step, we feed trans and formers to the model
and the label we wanted to predict is are.
This pattern continues, and as you can see,
the input sequence is the label sequence
just shifted by one.
Since the model only makes prediction after the first token,
the first element of the input sequence,
in this case, trans, is not used as a label.
Similarly, we don't have a label
for the last token in the sequence
since there is no token after the sequence ends.
Let's have a look at what we need to do
to create the labels for causal language modeling in code.
If we want to calculate a loss on a batch,
we can just pass the input_ids as labels
and all the shifting is handled in the model internally.
So, you see, there's no matching involved
in processing data for causal language modeling,
and it only requires a few simple steps.
(transition music)