(whooshing sound)
- How to instantiate the Transformers model?
In this video, we will look at how we can create
and use a model from the Transformers library.
As we've seen before, the TFAutoModel class
allows you to instantiate a pre-trained model
from any checkpoint on the Hugging Face Hub.
It will pick the right model class from the library
to instantiate the proper architecture
and load the weights of the pre-trained model inside.
As we can see, when given a BERT checkpoint,
we end up with a TFBertModel,
and similarly for GPT2 or BART.
Behind the scenes, this API can take the name
of a checkpoint on the Hub,
in which case it will download and cache
the configuration file as well as the model weights file.
You can also specify the path to a local folder
that contains a valid configuration file
and a model weights file.
To instantiate the pre-trained model,
the TFAutoModel API will first open the configuration file
to look at the configuration class that should be used.
The configuration class depends on the type of the model,
BERT, GPT2 or BART for instance.
Once it has the proper configuration class,
it can instantiate that configuration,
which is a blueprint to know how to create the model.
It also uses this configuration class
to find the proper model class,
which is combined with the loaded configuration
to load the model.
This model is not yet our pre-trained model
as it has just been initialized with random weights.
The last step is to load the weights
from the model file inside this model.
To easily load the configuration of a model
from any checkpoint or a folder
containing the configuration file,
we can use the AutoConfig class.
Like the TFAutoModel class,
it will pick the right configuration class from the library.
We can also use the specific class
corresponding to a checkpoint,
but we will need to change the code
each time we want to try a different model architecture.
As we said before, the configuration of a model
is a blueprint that contains all the information necessary
to create the model architecture.
For instance, the BERT model
associated with the bert-base-cased checkpoint
has 12 layers, a hidden size of 768,
and a vocabulary size of 28,996.
Once we have the configuration,
we can create a model that has the same architecture
as our checkpoint but is randomly initialized.
We can then train it from scratch like any TensorFlow model.
We can also change any part of the configuration
by using keyword arguments.
The second snippet of code instantiates
a randomly initialized BERT model
with 10 layers instead of 12.
Saving a model once it's trained or fine-tuned is very easy.
We just have to use the save_pretrained method.
Here, the model will be saved in a folder
named my-bert-model inside the current working directory.
Such a model can then be reloaded
using the from_pretrained method.
To run it to a projects model to the Hub,
check out the push (mumbles) video.
(whooshing sound)