- In this video, we will see how to debug an error
you encounter when running Trainer.train
As an example, we will use this script that finetunes
a bert model on the GLUE MNLI dataset.
Checkout the videos linked below
to see how we came to such a script.
Here we want to learn how to debug the problems in it.
Running the script gives us an error pretty quickly.
It happens at the line
where we feed the inputs to the model,
according to the traceback.
That tells us there is a problem there,
but the problem could come from many different causes.
To debug an error in a training,
you need to make sure each step of the training pipeline
works as intended.
This means checking that the inputs of your dataset
are correct,
you can batch them together,
feed them through the model to get a loss,
then compute the gradients of that loss
before performing an optimizer step.
So let's start by looking at the training dataset
this Trainer is using.
There is definitely a problem here.
We see texts and not number.
The error message was telling us the model
did not get input IDs
and we do not have those in the dataset indeed.
Looking back at our code,
we can see we made a mistake
and passed the wrong datasets to the Trainer.
So let's fix that and run again.
Now we have a new error.
Inspecting the traceback
tells us it happens when we try to create a batch,
specifically to group the features in a tensor.
We can confirm this by asking the Trainer to get us a batch
of the training data loader,
which reproduces the same error.
Either by inspecting the inputs or debugging,
we can then see they are not all of the same size.
This is because we have not passed a data collator
to do the padding to the Trainer
and didn't pad when preprocessing the data either.
Padding inside the Trainer is normally the default,
but only if you provide your tokenizer to the Trainer,
and we forgot to do that.
So let's fix the issue and run again.
This time we get a nasty CUDA error.
They are very difficult to debug because for one,
they put your kernel in a state that is not recoverable
so you have to restart your notebook from the beginning
and two, the traceback is completely useless for those.
Here the traceback tells us the error happens
when we do the gradient computation with loss.backward,
but as we will see later on that is not the case.
This is because everything that happens on the GPU
is done asynchronously.
When you execute the model call,
what the program does is just stacking that
in the queue of GPU,
then if the GPU didn't have any current job to do,
the work will start on the GPU at the same time
as the CPU moves to the next instruction.
Continuing with the extraction of the loss,
this is stacked into the GPU queue
while the CPU moves to the instruction loss.backward.
But the GPU still hasn't finished
the forward pass of the model
since all that took no time at all.
The CPU stops moving forward,
because loss.backward as an instruction telling it to wait
for the GPUs to be finished,
to make sure the gradients are correct.
When the GPU encounters an error,
it gives it back to the CPU with a cryptic message
who raises the error at the wrong place.
So to debug this, we will need to execute the next steps
of the training pipeline on the CPU.
It is very easy to do,
and we get a traceback we can trust this time.
As we said before,
the error actually happens during the forward pass
of the model,
and not loss.backward.
It's an index error.
With a bit of debugging,
we see we have labels ranging from 0 to 2,
so three different values,
but our outputs have a shape of batch size per 2.
It looks like our model has the wrong number of labels.
We can indeed confirm that,
and now that we know it's easy to fix it in the code
by adding num_labels=3 when we create the model.
Now the training script will run to completion.
We did not need it yet,
but here is how we would debug the next step
of the pipeline, gradient computation,
as well as the optimizer step.
With all of this, good luck debugging your own trainings!