(air whooshing)
- Some bugs in your code are very straightforward.
You try running it, you get a syntax error somewhere,
Python tells you exactly where, and you fix it.
This is great, it's simple and it's satisfying.
Sometimes, though, things crash
and the error is impossible to understand.
This happens a lot in machine learning for a few reasons,
you're working with big data structures,
you're using these big, complex libraries
with a lot of moving parts,
and also you're doing a lot of GPU computing,
and that in general is much more difficult to debug.
In Keras there's the additional problem
that your models are often compiled before execution,
which is great for performance
but it makes debugging them very difficult as well.
So, this is going to be a video about what to do
when you run into one of those nightmare bugs
and you just have no idea where to begin with fixing it.
So, to give you some intuitions for
the most common things that go wrong
and cause these weird issues,
and show you where to look for the sources of bugs
that you encounter, let's use this example script.
So, I'll show it to you here in two parts.
First, we do all our imports, we load a dataset,
we create our tokenizer and we tokenize the dataset.
Next, we convert our datasets to TensorFlow datasets,
so that's tf.data.Dataset,
and that's so that we can run fit on them,
and then we load our model from a pretrained checkpoint,
we compile it and we fit it with those datasets.
So, this seems straightforward enough,
it's similar to what we've done in the course before.
But beware, this is spooky code
and hides many dark and mysterious secrets.
So, what happens when we run it?
Well, it's not great.
So, we get this error message, but what does it mean?
We tried to train on our data, but we got no gradient?
It's pretty perplexing, I mean, how do we even begin
to debug not getting a gradient?
So, when the error you get doesn't immediately suggest
where the problem is, the best solution
is often to walk through things in sequence,
making sure at each stage that the outputs look right,
that everything looks okay at that point.
And, of course, that means the place to start
is always to check your data.
So, the best way to make sure
that the data you're giving the model is good,
is to grab a batch from the tf.data.Dataset
that your model is training on,
and that's because it's right at the end
of the data pipeline.
And so that means that if those outputs are good,
you're guaranteed that your data pipeline is working well.
So, we can do that by looping over the dataset
for one iteration and then breaking,
and that gives us a single batch.
So, what do we get when we inspect that batch?
We'll see that we're not getting any gradient
because we're not passing labels to Keras.
So, our labels are in the batch,
but they're a key in the input dictionary
and they're not a separate label as Keras expects,
so this is one of the most common issues you'll encounter
when training Transformers models with TensorFlow.
Our models can all compute loss internally,
but to use that loss for training
the labels need to be passed in the input dictionary,
where the model can see them.
This internal loss is the loss that we use
when we don't specify a loss when we call compile,
when we don't specify a loss argument.
So, Keras, on the other hand,
usually expects labels to be passed separately
from the input dictionary,
and not to be visible to the model,
and loss computations will usually fail
if you don't do that
So we need to choose one or the other,
either we use the model's internal loss
and keep the labels where they are,
or we keep using Keras losses
but we move the labels to the place Keras expects them.
So, or simplicity here, let's fix this issue
by using the model's internal losses,
and we do that by removing the loss argument
from the call to compile.
So, what happens if we try training now?
So we recompile with that, we call model.fit, what happens?
Well, it runs this time but now we get a loss of NaN.
So, that's not good, NaN means not a number
and it's not a good loss to have in general.
In fact, if we inspect our model now,
we'll see that not only are all the outputs NaN,
all the weights are NaN as well, as well as the loss.
So once a single NaN creeps into your computations,
it tends to spread, because it propagates from the loss
and once it's at the loss it's at the gradient,
it gets to the gradient,
and then once it's in the gradient
it enters the weight updates,
and then all your weight updates end up as NaN as well.
So NaN just completely destroyed our model here,
but where did it creep in first?
So to find out, we need to back to a point
before the model was destroyed,
we need to re-initialize the model
and look at the outputs for just the first batch.
And when we do that,
we see that NaN first appears in the loss,
but only in some samples.
So you can see this in more detail
in the accompanying section of the course notes,
I am moving fairly quickly here,
but we find that if we look at the labels,
the samples with a loss of NaN all have a label of two.
So this gives us a very strong clue,
if we check the model with model.config.num_labels,
we see that the model thinks there's only two labels,
but if we see a value of two,
that means there's at least three labels
because 0 is a label as well.
So we got a loss of NaN
because we got an "impossible" label in our label set,
and to fix that we need to go back and set the model
to expect the right number of labels,
so we can set num_labels=3
when we initialize the model but from_pretrained,
and now hopefully we can avoid this issue.
So, now we think our data is good and our model is good
and so training should work
but if we try running model.fit, we, well...
I mean, we do get a loss,
it is a number and it is going down
but it's not going down very quickly
and if we keep running this out,
we'll find that it stalls at a fairly high loss value.
So, what's going on?
Well, when things are mostly working,
but training is just slow or a bit odd,
that can often be a good time to look at your optimizer
and your training hyperparameters.
And this is where I want to mention
one of the most common sources of issues
when you're working with Keras,
you can name things like optimizers with strings,
so Keras supports that and it's very convenient,
but if you do that all of the options
get silently set to their default values.
So we specified our optimizer as Adam,
but in the process we invisibly got
the default learning rate, which is 1e-3,
or 10 to the power of -3.
So this learning rate is way too high
for training transformer models,
we should go back and specify the learning rate directly,
not using a string.
So, good values here are between 1e-5 and 1e-4
so let's split the difference and pick 5e-5.
So if you recompile with that,
you'll find that training actually works, at last.
The loss goes down efficiently
and it converges to a lower value.
So, again, I did go through this quite quickly
and I strongly recommend checking out the course notes
to see this in more detail,
and to experiment with the code yourself
and see what the errors look like
and how you can approach them,
but I hope I've given you here a quick summary
of the most common bugs
and maybe the most common debugging approaches
to dealing with them.
So, good luck, and remember to take plenty of breaks
if your code is giving you a hard time.
(air whooshing)