（过渡音乐） (transition music)
- 在这个视频中，我们来看看 - In this video, we take a look at the data processing
训练因果语言模型所必需的数据处理。 necessary to train causal language models.
因果语言建模是 Causal language modeling is the task
基于先前的词元预测下一个词元的任务。 of predicting the next token based on the previous ones.
因果语言建模的另一个术语 Another term for causal language modeling
是自回归建模。 is autoregressive modeling.
在这里的示例中， In the example that you can see here,
例如，下一个词元可以是 the next token could, for example,
是 NLP，也可能是机器学习。 be NLP or it could be machine learning.
因果语言模型的一个流行示例 A popular example of causal language models
是 GPT 系列模型。 is the GPT family of models.
训练 GPT 等模型， To train models such as GPT,
我们通常从大量文本文件组成的语料库开始。 we usually start with a large corpus of text files.
这些文件可以是从互联网上抓取的网页 These files can be webpages scraped from the internet
例如 Common Crawl 数据集 such as the Common Crawl dataset
或者它们可以是来自 GitHub 的 Python 文件， or they can be Python files from GitHub,
就像你在这里看到的一样。 like the ones you can see here.
作为第一步，我们需要词元化这些文件 As a first step, we need to tokenize these files
这样我们就可以将它们输入给模型。 such that we can feed them through the model.
在这里，我们将词元化的文本显示为不同长度的条， Here, we show the tokenized texts as bars of various length,
表明它们有些长一些有些短一些。 illustrating that they're shorter and longer ones.
这在处理文本时很常见。 This is very common when working with text.
但是，转换模型的上下文窗口有限 However, transform models have a limited context window
并根据数据源的不同， and depending on the data source,
词元化的文本可能 it is possible that the tokenized texts
比这个窗口长得多。 are much longer than this window.
在这种情况下，我们可以将序列 In this case, we could just truncate the sequences
截断为上下文长度， to the context length,
但这意味着在第一个上下文窗口之后 but this would mean that we lose everything
我们将失去一切。 after the first context window.
使用 return_overflowing_tokens 标志， Using the return overflowing token flag,
我们可以使用分词器来创建块 we can use the tokenizer to create chunks
其中每个块都是上下文长度的大小。 with each one being the size of the context length.
有时，如果没有足够的词元来填充它 Sometimes, it can still happen
仍然会出现 that the last chunk is too short
最后一块太短的情况。 if there aren't enough tokens to fill it.
在这种情况下，我们可以将其删除。 In this case, we can just remove it.
使用 return_length 关键字， With the return_length keyword,
我们还从分词器中获取每个块的长度。 we also get the length of each chunk from the tokenizer.
此函数显示准备数据集 This function shows all the steps
所必需的所有步骤。 necessary to prepare the dataset.
首先，我们用我刚才提到的标志 First, we tokenize the dataset
词元化数据集。 with the flags I just mentioned.
然后，我们遍历每个块 Then, we go through each chunk
如果它的长度与上下文长度匹配， and if it's length matches the context length,
我们将它添加到我们返回的输入中。 we add it to the inputs we return.
我们可以将此函数应用于整个数据集。 We can apply this function to the whole dataset.
此外，我们确保 In addition, we make sure
使用批处理并删除现有列。 that to use batches and remove the existing columns.
我们之所以需要删除现有的列， We need to remove the existing columns,
是因为我们可以为每个文本创建多个样本， because we can create multiple samples per text,
和数据集中的形状 and the shapes in the dataset
在那种情况下将不再匹配。 would not match anymore in that case.
如果上下文长度与文件长度相似， If the context length is of similar lengths as the files,
这种方法不再那么有效了。 this approach doesn't work so well anymore.
在这个例子中，样本 1 和 2 In this example, both sample 1 and 2
比上下文大小短 are shorter than the context size
并且按照之前的方法处理的话，将会丢弃它。 and will be discarded with the previous approach.
在这种情况下，最好先词元化每个样本 In this case, it is better to first tokenize each sample
而不去截断 without truncation
然后连接词元化后的样本 and then concatenate the tokenized samples
并且之间以字符串结尾或 EOS 词元结尾。 with an end of string or EOS token in between.
最后，我们可以按照上下文长度分块这个长序列， Finally, we can chunk this long sequence
我们不会丢失太多序列 with the context length and we don't lose too many sequences
因为它们太短了。 because they're too short anymore.
到目前为止，我们只介绍了 So far, we have only talked
因果语言建模的输入， about the inputs for causal language modeling,
但还没有提到监督训练所需的标签。 but not the labels needed for supervised training.
当我们进行因果语言建模时， When we do causal language modeling,
我们不需要输入序列的任何额外标签 we don't require any extra labels for the input sequences
因为输入序列本身就是标签。 as the input sequences themselves are the labels.
在这个例子中，当我们将词元 Trans 提供给模型时， In this example, when we feed the token Trans to the model,
我们要预测的下一个词元是 formers 。 the next token we wanted to predict is formers.
在下一步中，我们将 Trans 和 formers 提供给模型 In the next step, we feed trans and formers to the model
我们想要预测的标签是 are。 and the label we wanted to predict is are.
这种模式仍在继续，如你所见， This pattern continues, and as you can see,
输入序列是前移了一个位置的 the input sequence is the label sequence
标签序列。 just shifted by one.
由于模型仅在第一个词元之后进行预测， Since the model only makes prediction after the first token,
输入序列的第一个元素， the first element of the input sequence,
在本例中，就是 Trans，不会作为标签使用。 in this case, Trans, is not used as a label.
同样，对于序列中的最后一个词元 Similarly, we don't have a label
我们也没有标签 for the last token in the sequence
因为序列结束后没有词元。 since there is no token after the sequence ends.
让我们看看当需要在代码中为因果语言建模创建标签 Let's have a look at what we need to do
我们需要做什么操作。 to create the labels for causal language modeling in code.
如果我们想计算一批的损失， If we want to calculate a loss on a batch,
我们可以将 input_ids 作为标签传递 we can just pass the input_ids as labels
所有的转移都在模型内部处理。 and all the shifting is handled in the model internally.
所以，你看，在处理因果语言建模的数据时， So, you see, there's no matching involved
不涉及任何匹配， in processing data for causal language modeling,
它只需要几个简单的步骤。 and it only requires a few simple steps.
（过渡音乐） (transition music)