- Let's take a look at subword based tokenization.
Understanding why subword based tokenization is
interesting requires understanding the flaws
of word based and corrector based tokenization.
If you haven't seen the first videos
on word based and character based tokenization
we recommend you check them
out before looking at this video.
Subword based tokenization lies in between character based
and word based tokenization algorithms.
The idea is to find a middle ground
between very large vocabularies
a large quantity of out vocabulary tokens
and a loss of meaning across very similar words
for word based tokenizers and very long sequences
as well as less meaningful individual tokens.
For character based tokenizers.
These algorithms rely on the following principle.
Frequently used words should not be split
into smaller subwords
while rare words should be decomposed
into meaningful subwords.
An example is the word dog.
We would like to have our tokenizer to have a single ID
for the word dog rather
than splitting it into correctors D O and G.
However, when encountering the word dogs
we would like our tokenize to understand that at the root
this is still the word dog.
With an added S, that slightly changes the meaning
while keeping the original idea.
Another example is a complex word like tokenization
which can be split into meaningful subwords.
The root
of the word is token and -ization completes the root
to give it a slightly different meaning.
It makes sense to split the word
into two, token as the root of the word,
labeled as the start of the word
and ization as additional information labeled
as a completion of the word.
In turn, the model will now be able to make sense
of token in different situations.
It will understand that the word's token, tokens, tokenizing
and tokenization have a similar meaning and are linked.
It's will also understand that tokenization, modernization
and immunization, which all have the same suffixes
are probably used in the same syntactic situations.
Subword based tokenizers generally have a way to
identify which tokens are a start of word
and which tokens complete start of words.
So here token as the start
of a ward and hash hash ization as completion of award.
Here, the hash hash prefix indicates that ization is part
of award rather than the beginning of it.
The hash hash comes
from the BERT tokenizer based on the word piece algorithm.
Other tokenizes use other prefixes which can be
placed to indicate part of words
like in here or start of words instead.
There are a lot
of different algorithms that can be used
for subword tokenization
and most models obtaining state-of-the-art results
in English today
use some kind of subword tokenization algorithms.
These approaches help in reducing the vocabulary sizes
by sharing information across different words
having the ability to have prefixes
and suffixes understood as such.
They keep meaning across very similar words
by recognizing similar tokens, making them up.