(object whooshing)
- The tokenizer pipeline.
In this video, we'll look at how a tokenizer converts
raw texts to numbers,
that a Transformer model can make sense of,
like when we execute this code.
Here is a quick overview
of what happens inside the tokenizer object:
first, the text is split into tokens,
which are words, parts of words, or punctuation symbols.
Then the tokenizer adds potential special tokens
and converts each token to their unique respective ID
as defined by the tokenizer's vocabulary.
As we'll see, it doesn't quite happen in this order,
but doing it like this is better for understandings.
The first step is to split our input text into tokens.
We use the tokenize method for this.
To do that, the tokenizer may first perform some operations,
like lowercasing all words, then follow a set of rules
to split the result in small chunks of text.
Most of the Transformer models uses
a word tokenization algorithm, which means
that one given word can be split
in several tokens like tokenize here.
Look at the "Tokenization algorithms" video link below
for more information.
The # # prefix we see in front of ize is
a convention used by Bert to indicate
this token is not the beginning of the word.
Other tokenizers may use different conventions however:
for instance, ALBERT tokenizers will add a long underscore
in front of all the tokens
that added space before them, which is a convention shared
by all sentencepiece tokenizers.
The second step of the tokenization pipeline is
to map those tokens to their respective IDs
as defined by the vocabulary of the tokenizer.
This is why we need to download the file
when we instantiate a tokenizer
with the from_pretrained method.
We have to make sure we use the same mapping
as when the model was pretrained.
To do this, we use the convert_tokens_to_ids method.
We may have noticed
that we don't have the exact same results
as in our first slide, or not
as this looks like a list of random numbers anyway,
in which case, allow me to refresh your memory.
We had a the number at the beginning and a number
at the end that are missing, those are the special tokens.
The special tokens are added by the prepare_for_model method
which knows the indices of this token
in the vocabulary and just adds the proper numbers.
in the input IDs list.
You can look at the special tokens
and, more generally,
at how the tokenizer has changed your text,
by using the decode method
on the outputs of the tokenizer object.
As for the prefix for beginning
of words/ part of words, for special tokens vary depending
on which tokenizer you're using.
So that tokenizer uses CLS and SEP,
but the roberta tokenizer uses HTML-like anchors
and .
Now that you know how the tokenizer works,
you can forget all those intermediate methods,
and then you remember that you just have to call it
on your input texts.
The output of a tokenizer don't
just contain the input IDs, however.
To learn what the attention mask is,
check out the "Batch input together" video.
To learn about token type IDs,
look at the "Process pairs of sentences" video.
(object whooshing)