- The tokenization pipeline involves several steps
that converts raw text into numbers.
In this video, we will see what happens
during the pre-tokenization step.
The pre-tokenization operation is the operation performed
after the normalization of the text
and before the application of the tokenization algorithm.
This step consists in applying rules
that do not need to be learned
to perform a first division of the text.
Let's look at how several tokenizers
pre-tokenize in this example.
The gpt2 pre-tokenization divides the text on spaces
and some punctuation, but not on the apostrophe.
We also notice that spaces have been replaced
by capital G with a dot above.
Albert's pre-tokenization divides the text
at the level of spaces,
adds a space at the beginning of the sentence,
and replaces spaces with a special underscore.
Finally, Bert's pre-tokenization divides the text
at the level of punctuation and spaces.
But unlike the previous tokenizers,
spaces are not transformed
and integrated into tokens produced with this pre-tokenizer.
Through this three example,
we could observe the two main type of operation
brought by the pre-tokenization;
some change on the text
and the division of the string into tokens
that can be associated to words.
Finally, the backend tokenizer of the fast tokenizers
also allows to test the pre-tokenization operation
very easily, thanks to its pre_tokenize_str method.
We notice that the output of this operation
is composed of both tokens and offsets,
which allow to link the tokens to its position in the text
given in input of the method.
This operation defines the largest tokens
that can be produced by the tokenization,
or in those words, the barriers of the sub-tokens
which will be produced then.
And that's all for the characteristic
of the pre-tokenizers.