(air whooshing)
- In this video, we're going to see
how to load and fine tune a pre-trained model.
It's very quick.
And if you've watched our pipeline videos,
which I'll link below, the process is very similar.
This time, though, we're going to be using transfer learning
and doing some training ourselves,
rather than just loading a model and using it as is.
So to learn more about transfer learning,
head to the 'What is transfer learning?' video,
and we'll link that below as well.
But for now, let's look at this code.
To start, we pick which model we want to start with.
In this case, we're going to use the famous,
the original BERT,
as the foundation for our training today.
But what is this monstrosity line,
this 'TFAutoModelForSequenceClassification'?
What does that mean?
Well, the TF stands for TensorFlow.
And the rest means,
take a language model,
and stick a sequence classification head onto it
if it doesn't have one already.
So this line of code loads BERT,
which is a general purpose language model,
it loads at weights, architecture, and all
and then adds a new sequence classification head onto it
with randomly initialized weights.
So this method needs to know two things.
Firstly, it needs to know the name of the model
you wanted to load, the architecture and weights for.
And secondly, it needs to know how many classes
your problem has, because that will determine the size,
the number of neurons in the output head.
So if you want to follow along with the data
from our datasets videos, which I'll link below,
then you'll have two classes, positive and negative,
and thus num_labels equals two.
But what about this compile line?
Well, if you're familiar with Keras,
you've probably seen this already.
But if not, this is one of the core methods in Keras
that you're gonna see again, and again.
You always need to compile your model before you train it.
And compile needs to know two things.
Firstly, it needs to know the loss function,
which is what you're trying to optimize.
So here, we import the SparseCategoricalCrossentropy
loss function.
So that's a mouthful, but it's the standard loss function
for any neural network that's doing a classification task.
It basically encourages the network
to output large values for the right class,
and low values for the wrong classes.
Note that you can specify the loss function as a string,
like we did with the optimizer.
But there's a risk there,
there's a very common trap people fall into,
which is that by default, this loss assumes
the output is probabilities after a softmax layer.
But what our model has actually output
is the values before the softmax,
often called the logits, sometimes logits.
No one's quite sure how to pronounce that one.
But you probably seen these before
in the video about pipelines.
So if you get this wrong, your model won't train
and it'll be very annoying to figure out why.
In future videos, we're gonna see
how to use the model's internal loss computations,
so that you don't have to specify the loss yourself
and you don't have to worry about these details.
But for now, remember to set from_logits to true.
The second thing compile needs to know
is the optimizer you want.
In our case, we use adam,
which is sort of the standard optimizer
for deep learning these days.
The one thing you might want to change
is the learning rate.
And to do that, we'll need to import the actual optimizer
rather than just calling it by string.
But we'll talk about that in another video,
which I'll link below.
For now, let's just try training the model.
Well, so how do you train the model?
Again, if you've used Keras before,
this is all going to be very familiar to you.
But if not, let's very quickly look
at what we're doing here.
fit is pretty much the central method for Keras models.
It tells the model to train on the data we're passing in.
So here we pass the datasets we made
in the previous section,
the dataset contains both our inputs and our labels.
So we don't need to specify separate labels,
when we're calling fit.
Then we do the same thing with the validation_data.
And then we can if we want, we can specify details,
like the number of epochs for training
where there's some other arguments you can pass to fit.
But in the end, you just pass all of this to model.fit
and you let it run.
If everything works out,
you should see a little training bar
progressing along as your loss goes down.
And that's it.
While that's running,
you know, you can call your boss and tell them
you're a senior NLP machine learning engineer now
and you're gonna want a salary review next quarter.
These few lines of code are really all it takes
to apply the power of a massive
pre-trained language problem,
massive pre-trained language model, excuse me,
to your NLP problem.
But could we do better than this?
I mean, we certainly could.
With a few more advanced Keras features
like a tuned, scheduled learning rate,
we can get an even lower loss
and an even more accurate, more useful model.
And what do we do with our model after we train it?
So all of this is going to be covered in the videos
that are coming up, so stay tuned.
(air whooshing)