(intro striking)
- In this video, we'll study the encoder architecture.
An example of a popular encoder only architecture is BURT
which is the most popular model of its kind.
Let's first start by understanding how it works.
We'll use a small example using three words.
We use these as inputs and pass them through the encoder.
We retrieve a numerical representation of each word.
Here, for example, the encoder converts those three words,
welcome to NYC, in these three sequences of numbers.
The encoder outputs exactly one sequence of numbers
per input word.
This numerical representation can also be called
a feature vector, or a feature tensor.
Let's dive into this representation.
It contains one vector per word
that was passed through the encoder.
Each of these vector is a numerical representation
of the word in question.
The dimension of that vector is defined
by the architecture of the model.
For the base BERT model, it is 768.
These representations contain the value of a word,
but contextualized.
For example, the vector attributed to the word "to"
isn't the representation of only the "to" word.
It also takes into account the words around it
which we call the context.
As in it looks to the left context,
the words on the left of the one we're studying,
here the word "Welcome",
and the context on the right, here the word "NYC",
and it outputs a value for the word given its context.
It is therefore a contextualized value.
One could say that the vector of 768 values
holds the meaning of the word within the text.
It does this thanks to the self-attention mechanism.
The self-attention mechanism relates to different positions,
or different words in a single sequence
in order to compute a representation of that sequence.
As we've seen before, this means that
the resulting representation of a word
has been affected by other words in the sequence.
We won't dive into the specifics here
which will offer some further readings
if you want to get a better understanding
at what happens under the hood.
So why should one use and encoder?
Encoders can be used as stand-alone models
in a wide variety of tasks.
For example, BERT,
arguably the most famous transformer model,
is a standalone encoder model,
and at the time of release,
it'd be the state of the art
in many sequence classification tasks,
question answering tasks,
and mask language modeling to only cite of few.
The idea is that encoders are very powerful
at extracting vectors that carry
meaningful information about a sequence.
This vector can then be handled down the road
by additional neurons to make sense of them.
Let's take a look at some examples
where encoder really shine.
First of all, Masked Language Modeling, or MLM.
It's the task of predicting a hidden word
in a sequence of word.
Here, for example, we have hidden the word
between "My" and "is".
This is one of the objectives with which BERT was trained.
It was trained to predict hidden words in a sequence.
Encoders shine in this scenario in particular
as bi-directional information is crucial here.
If we didn't have the words on the right,
"is", "Sylvain" and the ".",
then there is very little chance
that BERT would have been able to identify name
as the correct word.
The encoder needs to have a good understanding
of the sequence in order to predict a masked word
as even if the text is grammatically correct,
it does not necessarily make sense
in the context of the sequence.
As mentioned earlier,
encoders are good at doing sequence classification.
Sentiment analysis is an example of sequence classification.
The model's aim is to identify the sentiment of a sequence.
It can range from giving a sequence,
a rating from one to five stars
if doing review analysis to giving a positive,
or negative rating to a sequence
which is what is shown here.
For example, here, given the two sequences,
we use the model to compute a prediction,
and to classify the sequences
among these two classes, positive and negative.
While the two sequences are very similar
containing the same words,
the meaning is entirely different,
and the encoder model is able to grasp that difference.
(outro striking)