(air whooshing)
- In our other videos,
and as always, there'll be links below
if you want to check those out,
we showed you how to initialize and fine-tune
a transformer model in TensorFlow.
So the question now is, what can we do with a model
after we train it?
The obvious thing to try
is to use it to get predictions for new data,
so let's see how to do that.
Again, if you're familiar with Keras, the good news
is that because there are just standard Keras models,
we can use the standard Keras predict method,
as shown here.
You simply pass in tokenized text to this method,
like you'd get from a tokenizer,
and you get your results.
Our models can output several different things,
depending on the options you set,
but most of the time the thing you want
is the output logits.
If you haven't come across them before logits,
sometimes pronounced to logits because no one's sure,
are the outputs of the last layer of the network
because before a softmax has been applied.
So if you want to turn the logits
into the model's probability outputs,
you just apply a softmax, like so.
What if we want to turn those probabilities
into class predictions?
Again, it's very straightforward.
We just pick the biggest probability for each output
and you can get that immediately with the argmax function.
argmax will return the index
of the largest probability in each row
which means that we'll get a vector of integers.
So zero if the largest probability was in the zero position,
one in the first position, and so on.
So these are our class predictions indicating class zero,
class one, and so on.
In fact, if class predictions are all you want,
you can skip the softmax step entirely
because the largest logit will always be the largest
probability as well.
So if probabilities and class predictions are all you want,
then you've seen everything you need at this point.
But if you're interested in benchmarking your model
or using it for research,
you might want to delve deeper into the results you get.
And one way to do that is to compute some metrics
for the model's predictions.
If you're following along with our datasets
and fine tuning videos,
we got our data from the MRPC dataset,
which is part of the GLUE benchmark.
Each of the GLUE datasets
as well as many other datasets in our dataset, Light Hub
has some predefined metrics,
and we can load them easily
with the datasets load metric function.
For the MRPC dataset, the built-in metrics are accuracy
which just measures the percentage of the time
the model's prediction was correct,
and the F1 score,
which is a slightly more complex measure
that measures how well the model trades off
precision and recall.
To compute those metrics to benchmark our model,
we just pass them the model's predictions,
and to the ground truth labels,
and we get our results in a straightforward Python dict.
If you're familiar with Keras though,
you might notice that this is a slightly weird way
to compute metrics,
because we're only computing metrics
at the very end of training.
But in Keras, you have this built-in ability
to compute a wide range of metrics
on the fly while you're training,
which gives you a very useful insight
into how training is going.
So if you want to use built-in metrics,
it's very straightforward
and you use the standard Keras approach again.
You just pass a metric argument to the compile method.
As with things like loss and optimizer,
you can specify the metrics you want by string
or you can import the actual metric objects
and pass specific arguments to them.
But note that unlike loss and accuracy,
you have to supply metrics as a list
even if there's only one metric you want.
Once a model has been compiled with a metric,
it will report that metric for training,
validation, and predictions.
Assuming there are labels passed to the predictions.
You can even write your own metric classes.
Although this is a bit beyond the scope of this course,
I'll link to the relevant TF docs below
because it can be very handy
if you want a metric that isn't supported
by default in Keras,
such as the F1 score.
(air whooshing)