- The fast tokenizers
of the Transformers library are fast,
but they also implement features that will be super useful
for data pre-processing and post-processing.
Let's have a look at them!
First, let's have a look
at the usual output of a tokenizer.
We get input IDs that correspond to token,
but we lose a lot of information in the process.
For instance,
here the tokenization is the same for the two sentences
even if one has several more spaces than the other.
Just having the input IDs is thus not enough
if we want to match some tokens with a span of text,
something we'll need to do
when tackling question answering, for instance.
It's also difficult to know
when two tokens belong to the same word or not.
It looks easy when you just look at the output
of a BERT tokenizer where we just need to look
for the hash hash.
But other tokenizers have different ways
to tokenize parts of words.
For instance, RoBERTa adds this special G symbol
to mark the tokens at the beginning of the word
and T5 uses this special underscore symbol
for the same purpose.
Thankfully, the fast tokenizers keep track of the word
each token comes from,
with a word_ids method you can use on their outputs.
The output is not necessarily clear,
but assembled together in a nice table like this,
we can look at the word position for each token.
Even better, the fast tokenizers keep track
of the span of characters each token comes from,
and we can get them when calling it on one
or several text by adding
the return_offsets_mapping=True argument.
In this instance, we can see how we jump positions
between the hash hash token and the super token,
because of the multiple spaces in the initial sentence.
To enable this, the fast tokenizers
store additional information at each step
of their internal pipeline.
That internal pipeline consists of normalization,
where we apply some cleaning to the text,
like lower casing or removing the accents;
pre-tokenization,
which is where we split the texts into words;
then we apply the model of the tokenizer,
which is where the words are split into tokens,
before finally doing the post processing,
where special tokens are added.
From the beginning to the end of the pipeline,
the tokenizer keeps track of each span of text
that corresponds to each word, then each token.
We'll see how useful it is
when we tackle the following tasks:
when doing masked language modeling
one variation that gets state-of-the-art results
is to mask all the tokens of a given word
instead of randomly chosen words.
This will require us to use the word IDs we saw.
When doing token classification,
we'll need to convert the labels we have on words,
to labels on each tokens.
As for the offset mappings,
it will be super useful when we need to convert
token positions in a sentence into a span of text,
which we'll need to know when we're looking
at question answering
or when grouping the tokens corresponding
to the same entity in token classification.
To have a look at these tasks,
check the videos linked below!