(whooshing sound)
- Let's have a look
inside the token classification pipeline.
In the pipeline video,
we looked at the different applications
the Transformers library supports out of the box.
One of them being token classification.
For instance, predicting for each word in a sentence,
whether they correspond to a person,
an organization, or location.
We can even group together the tokens
corresponding to the same entity.
For instance, all the tokens that form the word Sylvain here
or Hugging and Face.
So, token classification pipeline
works the same way as a text classification pipeline
we studied in a previous video.
There are three steps.
Tokenization, the model, and the post processing.
The first two steps are identical
to the text classification pipeline,
except we use an auto token classification model
instead of a sequence classification one.
We tokenize our text, then feed it to the model.
Instead of getting one number for each possible level
for the whole sentence,
we get one number for each of the possible nine levels
for every token in the sentence.
Here, 19.
Like all the other models of the Transformers library,
our model outputs logits which we need to turn
into predictions by using a SoftMax.
We also get the predicted label for each token
by taking the maximum prediction.
Since the softmax function preserves the order,
we could have done it on the logits
if we had no need of the predictions.
The model config contains the label mapping
in its id2label field.
Using it, we can map every token to its corresponding label.
The label O corresponds to "no entity"
which is why we didn't see it in our results
in the first slide.
On top of the label and the probability,
those results included the start
and end character in the sentence.
We'll need to use the offset mapping
of the tokenizer to get those.
Look at the video link below
if you don't know about them already.
Then, looping through each token
that has a label distinct from O,
we can build the list of results
we got with our first pipeline.
The last step is to group together tokens
that corresponds to the same entity.
This is why we had two labels for each type of entity,
I-PER and B-PER for instance.
It allows us to know if a token
is in the same entity as a previous one.
Note that there are two ways
of labeling used for token classification.
One, in pink here, uses the B-PER label
at the beginning of each new entity.
But the other in blue only uses it
to separate two adjacent entities of the same types.
In both cases we can flag a new entity
each time we see a new label appearing,
either with the I or B prefix.
Then, take all the following tokens labeled the same
with an I-flag.
This, coupled with the offset mapping
to get the start and end characters
allows us to get the span of texts for each entity.
(whooshing sound)