- 在本视频中，我们将了解如何调试错误 - In this video, we will see how to debug an error
你在运行 Trainer.train 时你会遇到的 you encounter when running Trainer.train
作为一个例子，我们将使用这个脚本微调 As an example, we will use this script that finetunes
一个 BERT 模型, 在 GLUE MNLI 数据集上。 a BERT model on the GLUE MNLI dataset.
查看下面链接的视频 Checkout the videos linked below
以看看我们是如何得出这样一个脚本的。 to see how we came to such a script.
这里我们要学习如何调试其中的问题。 Here we want to learn how to debug the problems in it.
运行脚本很快就会给我们一个错误。 Running the script gives us an error pretty quickly.
它发生在这一行 It happens at the line
我们将输入提供给模型， where we feed the inputs to the model,
根据回溯。 according to the traceback.
这告诉我们这里有问题， That tells us there is a problem there,
但问题可能有许多不同的原因。 but the problem could come from many different causes.
要调试训练中的错误， To debug an error in a training,
你需要确保训练 pipeline 的每一步 you need to make sure each step of the training pipeline
按预期工作。 works as intended.
这意味着检查数据集的输入 This means checking that the inputs of your dataset
是正确的， are correct,
你可以把它们分批在一起， you can batch them together,
把他们通过模型以获得损失， feed them through the model to get a loss,
然后计算该损失的梯度 then compute the gradients of that loss
在执行优化器步骤之前。 before performing an optimizer step.
因此，让我们从查看训练数据集开始 So let's start by looking at the training dataset
这个 Trainer 正在使用。 this Trainer is using.
这里肯定有问题。 There is definitely a problem here.
我们看到的是文字而不是数字。 We see texts and not number.
错误消息告诉我们模型 The error message was telling us the model
没有得到输入 ID did not get input IDs
我们数据集中确实没有那些。 and we do not have those in the dataset indeed.
回顾我们的代码， Looking back at our code,
我们可以看到我们犯了一个错误 we can see we made a mistake
并将错误的数据集传递给 Trainer 。 and passed the wrong datasets to the Trainer.
所以让我们修复它并再次运行。 So let's fix that and run again.
现在我们有一个新的错误。 Now we have a new error.
检查回溯 Inspecting the traceback
告诉我们当我们尝试创建批处理时会发生， tells us it happens when we try to create a batch,
特别对于对张量中的特征进行分组。 specifically to group the features in a tensor.
我们可以通过要求 Trainer 给我们一分批来确认这一点 We can confirm this by asking the Trainer to get us a batch
训练数据加载器， of the training data loader,
复现相同的错误。 which reproduces the same error.
通过检查输入或调试， Either by inspecting the inputs or debugging,
然后我们可以看到它们的大小并不相同。 we can then see they are not all of the same size.
这是因为我们还没有输入数据整理器 This is because we have not passed a data collator
对 Trainer 进行填充 to do the padding to the Trainer
并且在预处理数据时也没有填充。 and didn't pad when preprocessing the data either.
Trainer 内部的填充通常是默认设置， Padding inside the Trainer is normally the default,
但前提是你将 tokenizer 提供给 Trainer ， but only if you provide your tokenizer to the Trainer,
我们忘了这样做。 and we forgot to do that.
因此，让我们解决问题并再次运行。 So let's fix the issue and run again.
这次我们得到了一个讨厌的 CUDA 错误。 This time we get a nasty CUDA error.
它们很难调试，因为一方面， They are very difficult to debug because for one,
他们将你的内核置于不可恢复的状态 they put your kernel in a state that is not recoverable
所以你必须从头开始重启你的笔记本 so you have to restart your notebook from the beginning
第二，追溯对那些来说完全没用。 and two, the traceback is completely useless for those.
这里的回溯告诉我们错误发生了 Here the traceback tells us the error happens
当我们使用 loss.backward 进行梯度计算时， when we do the gradient computation with loss.backward,
但正如我们稍后将看到的那样，情况并非如此。 but as we will see later on that is not the case.
这是因为在 GPU 上发生的一切 This is because everything that happens on the GPU
是异步完成的。 is done asynchronously.
当你执行模型调用时， When you execute the model call,
该程序所做的只是堆叠它 what the program does is just stacking that
在 GPU 的队列中， in the queue of GPU,
那么如果 GPU 当前没有任何工作要做， then if the GPU didn't have any current job to do,
工作将同时在 GPU 上开始 the work will start on the GPU at the same time
当 CPU 移动到下一条指令时。 as the CPU moves to the next instruction.
继续提取损失， Continuing with the extraction of the loss,
这被堆叠到 GPU 队列中 this is stacked into the GPU queue
同时 CPU 移动到指令 loss.backward。 while the CPU moves to the instruction loss.backward.
但是 GPU 还没有完成 But the GPU still hasn't finished
模型的前向传播 the forward pass of the model
因为这一切根本不需要时间。 since all that took no time at all.
CPU 停止前进， The CPU stops moving forward,
因为 loss.backward 作为指令告诉它等待 because loss.backward as an instruction telling it to wait
为了完成 GPU， for the GPUs to be finished,
以确保梯度是正确的。 to make sure the gradients are correct.
当 GPU 遇到错误时， When the GPU encounters an error,
它通过一条神秘的消息将它返回给 CPU it gives it back to the CPU with a cryptic message
谁在错误的地方提出错误。 who raises the error at the wrong place.
所以要调试这个，我们需要执行 So to debug this, we will need to execute
接下来 CPU 上的训练流水线的步骤。 the next steps of the training pipeline on the CPU.
这很容易做到， It is very easy to do,
这次我们得到了可以信任的回溯。 and we get a traceback we can trust this time.
正如我们之前所说， As we said before,
错误实际上发生在前向传播过程中 the error actually happens during the forward pass
对模型， of the model,
而不是 loss.backward。 and not loss.backward.
这是一个索引错误。 It's an index error.
经过一些调试， With a bit of debugging,
我们看到我们有从 0 到 2 的标签， we see we have labels ranging from 0 to 2,
所以三个不同的值， so three different values,
但我们的输出具有每 2 个批量大小的形状。 but our outputs have a shape of batch size per 2.
看起来我们的模型有错误的标签数量。 It looks like our model has the wrong number of labels.
我们确实可以确认， We can indeed confirm that,
现在我们知道在代码中修复它很容易 and now that we know it's easy to fix it in the code
通过在创建模型时添加 num_labels=3。 by adding num_labels=3 when we create the model.
现在训练脚本将运行完成。 Now the training script will run to completion.
我们还不需要它， We did not need it yet,
但这是我们调试下一步的方式 but here is how we would debug the next step
pipeline ，梯度计算， of the pipeline, gradient computation,
以及优化器更新。 as well as the optimizer step.
有了所有这些，祝你在调试自己的训练时好运！ With all of this, good luck debugging your own trainings!