(logo whooshing)
- What happens inside the pipeline function?
In this video, we will look at what actually happens
when we use the pipeline function
of the Transformers library.
More specifically, we will look
at the sentiment analysis pipeline,
and how it went from the two following sentences,
to the positive and negative labels
with their respective scores.
As we have seen in the pipeline presentation,
there are three stages in the pipeline.
First, we convert the raw texts to numbers
the model can make sense of using a tokenizer.
Then those numbers go through the model,
which outputs logits.
Finally, the post-processing steps transforms
those logits into labels and scores.
Let's look in detail at those three steps
and how to replicate them using the Transformers library,
beginning with the first stage, tokenization.
The tokenization process has several steps.
First, the text is split into small chunks called tokens.
They can be words, parts of words or punctuation symbols.
Then the tokenizer will had some special tokens,
if the model expect them.
Here the model uses expects a CLS token at the beginning
and a SEP token at the end of the sentence to classify.
Lastly, the tokenizer matches each token to its unique ID
in the vocabulary of the pretrained model.
To load such a tokenizer,
the Transformers library provides the AutoTokenizer API.
The most important method of this class is from_pretrained,
which will download and cache the configuration
and the vocabulary associated to a given checkpoint.
Here the checkpoint used by default
for the sentiment analysis pipeline
is distilbert-base-uncased-finetuned-sst-2-English.
(indistinct)
We instantiate a tokenizer associated with that checkpoint,
then feed it the two sentences.
Since those two sentences are not of the same size,
we will need to pad the shortest one
to be able to build an array.
This is done by the tokenizer with the option, padding=True.
With truncation=True, we ensure that any sentence
longer than the maximum the model can handle is truncated.
Lastly, the return_tensors option
tells the tokenizer to return a PyTorch tensor.
Looking at the result, we see we have a dictionary
with two keys.
Input IDs contains the IDs of both sentences,
with zero where the padding is applied.
The second key, attention mask,
indicates where padding has been applied,
so the model does not pay attention to it.
This is all what is inside the tokenization step.
Now, let's have a look at the second step, the model.
As for the tokenizer,
there is an AutoModel API with a from_pretrained method.
It will download and cache the configuration of the model
as well as the pretrained weights.
However, the AutoModel API
will only instantiate the body of the model,
that is the part of the model that is left
once the pretraining head is removed.
It will output a high-dimensional tensor
that is a representation of the sentences passed,
but which is not directly useful
for our classification problem.
Here the tensor has two sentences, each of 16 tokens,
and the last dimension is the hidden size of our model, 768.
To get an output linked to our classification problem,
we need to use the AutoModelForSequenceClassification class.
It works exactly as the AutoModel class,
except that it will build a model
with a classification head.
There is one auto class for each common NLP task
in the Transformers library.
Here after giving our model the two sentences,
we get a tensor of size two by two,
one result for each sentence and for each possible label.
Those outputs are not probabilities yet,
we can see they don't sum to 1.
This is because each model
of the Transformers library returns logits.
To make sense of those logits,
we need to dig into the third and last step of the pipeline.
Post-processing, to convert logits into probabilities,
we need to apply a SoftMax layers to them.
As we can see,
this transforms them into positive number
that sum up to one.
The last step is to know which of those corresponds
to the positive or the negative label.
This is given by the id2label field of the model config.
The first probabilities, index zero,
correspond to the negative label,
and the seconds, index one,
correspond to the positive label.
This is how our classifier built
with the pipeline function picked those labels
and computed those scores.
Now that you know how each steps works,
you can easily tweak them to your needs.
(logo whooshing)