(screen swishing)
(text swishing)
In our other videos,
we talked about the basics
of fine-tuning a language model with Tensorflow,
and as always, when I refer to videos I'll link them below.
Still, can we do better?
So here's the code from our model fine-tuning video,
and while it works,
we could definitely tweak a couple of things.
By far the most important thing is the learning rate.
In this video we'll talk about how to change it,
which will make your training
much more consistently successful.
In fact, really there are two things
we want to change about the default learning rate for Adam.
So the first we want to change
is that it's way too high for our models,
by default, Adam uses a learning rate
of 10 to the minus 3, 1 E minus 3,
and that's very high for training transformer models.
We're going to start at 5 by 10 to the minus 5,
5 E minus 5, which is 20 times lower than the default.
And secondly, we don't just want a constant learning rate,
we can get even better performance
if we decay the learning rate down to a tiny value,
or even to zero , over the course of training.
So that's what this thing here,
this Polynomial Decay schedule thing is doing.
So I'll show you what that decay looks like in a second,
but first we need to tell the scheduler
how long training is going to be,
so that it decays at the right speed,
and that's what this code here is doing.
We're computing how many minibatches
the model is going to see over its entire training run,
which is the size of the training set,
and then we multiply that
by the number of epochs
to get the total number of batches
across the whole training run.
Once we know how many training steps we're taking,
we just pass all that information to the scheduler
and we're ready to go.
What does the polynomial decay schedule look like?
Well, it looks like this,
it starts at 5 E minus 5,
which means 5 times 10 to the minus 5,
and then decays down at a constant rate
until it hits zero right at the very end of training.
So hold on, I can already hear you
yelling at your monitor, though,
and yes, I know,
this is actually constant or a linear decay,
and I know the name is polynomial,
and you're feeling cheated that, you know,
you were promised a polynomial and haven't gotten it,
so calm down though, it's okay,
because, of course, linear functions are just
first-order special cases
of the general polynomial functions,
and if you tweak the options to this class,
you can get a truly polynomial,
a higher-order decay schedule,
but this linear schedule will work fine for us for now,
we don't actually need all those
fancy tweaks and fancy gadgets.
So coming back,
how do we actually use this learning rate schedule
once we've created it?
So it's simple, we just pass it to Adam.
So the first time we compiled the model,
we just passed the string Adam,
to get our optimizer.
So Keras recognizes the names of common optimizers
and loss functions if you pass them as strings,
so it saves time to do that
if you only want the default settings.
But now we're professional machine learners,
and, you know, that salary review is upcoming,
so we've got our very own learning rate schedule,
and we're gonna do things properly.
So the first we do is we import the optimizer,
and then we initialize it with a scheduler,
which is getting passed to to the learning rate argument
of that optimizer.
And now we compile the model with this new optimizer,
and again, whatever loss function you want,
so this is going to be sparse categorical crossentropy
if you're following along from the fine-tuning video.
And then, we're we're ready to go,
now we have a high-performance model,
and ready for training.
All that remains is to fit the model
just like we did before.
Remember, because we compiled the model
with the new optimizer and the new learning rate schedule,
we actually don't need to change anything at all
when we call fit,
we just call it again,
with exactly the same command as before,
but now we get a beautiful training,
with a nice, smooth learning rate decay,
starting from a good value,
and decaying down to zero.
(screen swishing)