(air whooshing)
- Let's see together what is the training strategy
of the WordPiece algorithm, and how it performs
the tokenization of a text, once trained.
WordPiece is a tokenization algorithm introduced by Google.
It is used, for example, by BERT.
To our knowledge,
the code of WordPiece has not been open source.
So we base our explanations
on our own interpretation of the published literature.
So, what is the training strategy of WordPiece?
Similarly to the BPE algorithm,
WordPiece starts by establishing an initial vocabulary
composed of elementary units,
and then increases this vocabulary to the desired size.
To build the initial vocabulary,
we divide each word in the training corpus
into the sequence of letters that make it up.
As you can see, there is a small subtlety.
We add two hashtags in front of the letters
that do not start a word.
By keeping only one occurrence per elementary unit,
we now have our initial vocabulary.
We will list all the existing pairs in our corpus.
Once we have this list,
we will calculate a score for each of these pairs.
As for the BPE algorithm,
we will select the pair with the highest score.
Taking for example,
the first pair composed of the letters H and U.
The score of a pair is simply equal to the frequency
of appearance of the pair, divided by the product
of the frequency of appearance of the first token,
by the frequency of appearance of the second token.
Thus, at a fixed frequency of appearance of the pair,
if the subparts of the pair are very frequent in the corpus,
then this score will be decreased.
In our example, the pair HU appears four times,
the letter H four times, and the letter U four times.
This gives us a score of 0.25.
Now that we know how to calculate this score,
we can do it for all pairs.
We can now add to the vocabulary
the pair with the highest score, after merging it of course.
And now we can apply this same fusion to our split corpus.
As you can imagine,
we just have to repeat the same operations
until we have the vocabulary at the desired size.
Let's look at a few more steps
to see the evolution of our vocabulary,
and also the evolution of the length of the splits.
And now that we are happy with our vocabulary,
you are probably wondering how to use it to tokenize a text.
Let's say we want to tokenize the word "huggingface".
WordPiece follows these rules.
We will look for the longest possible token
at the beginning of the word.
Then we start again on the remaining part of our word,
and so on until we reach the end.
And that's it. Huggingface is divided into four sub-tokens.
This video is about to end.
I hope it helped you to understand better
what is behind the work, WordPiece.
(air whooshing)