(logo swooshes)
- How to instantiate a Transformers model.
In this video, we'll look at how we can create a user model
from the Transformers library.
As we have seen before the AutoModel class allows
you to instantiate a pretrained model
from any checkpoint on the Hugging Face Hub.
It'll pick the right model class
from the library to instantiate the proper architecture
and loads of weights as the pretrained model inside.
As we can see, when given a BERT checkpoint
we end up with a BertModel and similarly, for GPT-2 or BART.
Behind the scenes,this API can take the name
of a checkpoint on the Hub
in which case it will download and cache the configuration
file as well as a model weights file.
You can also specify the path to a local folder
that contains a valid configuration file and a
model of weights file.
To instantiate the pretrained model,
the AutoModel API will first open the configuration
file to look at a configuration class that should be used.
The configuration class depends on the type
of the model BERT, GPT-2 or BART for instance.
Once it has a proper configuration class,
it can instantiate that configuration
which is a blueprint to know how to create the model.
It also uses this configuration class to
find the proper model class, which is then combined
with the loaded configuration to load the model.
This model is not yet a pretrained model
as it has just been initialized with random weights.
The last step is to load the weight from the model file
inside this model.
To easily load the configuration of a model
from any checkpoint or folder containing
the configuration file.
We can use the AutoConfig class.
Like the AutoModel class,
it will pick the right configuration class from the library.
We can also use a specific class corresponding
to a checkpoint, but we will need to change
the code each time we want to try
a different model architecture.
As we said before, the configuration
of a model is a blueprint that contains all the
information necessary to create the model architecture.
For instance, the BERT model associated
with the bert-base-cased checkpoint has 12 layers,
a hidden side of 768 and a vocabulary side of 28,996.
Once we have the configuration,
we can create a model that does the same architecture
as our checkpoint, but is randomly initialized.
We can then train it from scratch.
Like any bio PyTorch module
We can also change any part
of the configuration by using keyword arguments.
The second snippet of code instantiates
a randomly initialized BERT model
with 10 layers instead of 12.
Saving a model once it's trained or fine-tuned is very easy.
We just have to use a safe pretrained method.
Here the model will be saved in a folder named
"my-bert-model" inside the current working directory.
Such a model can then be reloaded using the form
pretrained method.
To learn how to easily approach this model
to that, check out the push to a video.