(air whooshing)
In this video, we will see how
you can create your own tokenizer from scratch.
To create your own tokenizer, you will have to think about
each of the operations involved in tokenization.
Namely, the normalization, the pre-tokenization,
the model, the post processing, and the decoding.
If you don't know what normalization,
pre-tokenization, and the model are,
I advise you to go and see the videos linked below.
The post processing gathers all the modifications
that we will carry out on the tokenized text.
It can include the addition of special tokens,
the creation of an intention mask,
but also the generation of a list of token IDs.
The decoding operation occurs at the very end,
and will allow passing
from the sequence of IDs in a sentence.
For example, you can see that the hashtags
have been removed, and the tokens
composing the word today have been grouped together.
In a fast tokenizer, all these components
are gathered in the backend_tokenizer attribute.
As you can see with this small code snippet,
it is an instance of a tokenizer
from the tokenizers library.
So, to create your own tokenizer,
you will have to follow these steps.
First, create a training dataset.
Second, create and train a tokenizer
with the transformer library.
And third, load this tokenizer into a transformer tokenizer.
To understand these steps,
I propose that we recreate a BERT tokenizer together.
The first thing to do is to create a dataset.
With this code snippet you can create an iterator
on the dataset wikitext-2-raw-V1,
which is a rather small dataset in English,
perfect for the example.
We attack here the big part,
the design of our tokenizer with the tokenizer library.
We start by initializing a tokenizer instance
with a WordPiece model because it is the model used by BERT.
Then we can define our normalizer.
We will define it as a succession of two normalizations
used to clean up characters not visible in the text.
One lowercasing normalization,
and two last normalizations used to remove accents.
For the pre-tokenization, we will chain two pre_tokenizers.
The first one separating the text at the level of spaces,
and the second one isolating the punctuation marks.
Now, we can define the trainer that will allow us
to train the WordPiece model chosen at the beginning.
To carry out the training,
we will have to choose a vocabulary size.
Here we choose 25,000.
And we also need to announce the special tokens
that we absolutely want to add to our vocabulary.
In one line of code, we can train our WordPiece model
using the iterator we defined earlier.
Once the model has been trained, we can retrieve
the IDs of the special class and separation tokens,
because we will need them to post-process our sequence.
Thanks to the TemplateProcessing class,
we can add the CLS token at the beginning of each sequence,
and the SEP token at the end of the sequence,
and between two sentences if we tokenize a pair of text.
Finally, we just have to define our decoder,
which will allow us to remove the hashtags
at the beginning of the tokens
that must be reattached to the previous token.
And there it is.
You have all the necessary lines of code
to define your own tokenizer with the tokenizer library.
Now that we have a brand new tokenizer
with the tokenizer library, we just have to load it
into a fast tokenizer from the transformers library.
Here again, we have several possibilities.
We can load it in the generic class,
PreTrainedTokenizerFast, or in the BertTokenizerFast class
since we have built a BERT like tokenizer here.
I really hope this video has helped you understand
how you can create your own tokenizer,
and that you are ready now to navigate
the tokenizer library documentation
to choose the components for your brand new tokenizer.
(air whooshing)