(page whirring)
- Before diving in character-based tokenization,
understanding why this kind of tokenization is interesting
requires understanding the flaws of word-based tokenization.
If you haven't seen the first video
on word-based tokenization
we recommend you check it out before looking at this video.
Okay, let's take a look at character-based tokenization.
We now split our text into individual characters,
rather than words.
There are generally a lot of different words in languages,
while the number of characters stays low.
To begin let's take a look at the English language,
it has an estimated 170,000 different words,
so we would need a very large vocabulary
to encompass all words.
With a character-based vocabulary,
we can get by with only 256 characters,
which includes letters, numbers and special characters.
Even languages with a lot of different characters
like the Chinese languages can have dictionaries
with up to 20,000 different characters
but more than 375,000 different words.
So character-based vocabularies
let us use fewer different tokens
than the word-based tokenization dictionaries
we would otherwise use.
These vocabularies are also more complete
than their word-based vocabularies counterparts.
As our vocabulary contains all characters
used in a language, even words unseen
during the tokenizer training can still be tokenized,
so out-of-vocabulary tokens will be less frequent.
This includes the ability to correctly tokenize
misspelled words, rather than discarding them
as unknown straight away.
However, this algorithm isn't perfect either.
Intuitively, characters do not hold
as much information individually as a word would hold.
For example, "Let's" holds more information
than it's first letter "l".
Of course, this is not true for all languages,
as some languages like ideogram-based languages
have a lot of information held in single characters,
but for others like roman-based languages,
the model will have to make sense of multiple tokens
at a time to get the information otherwise held
in a single word.
This leads to another issue with character-based tokenizers,
their sequences are translated into very large amount
of tokens to be processed by the model.
And this can have an impact on the size of the context
the model will carry around, and will reduce the size
of the text we can use as input for our model,
which is often limited.
This tokenization, while it has some issues,
has seen some very good results in the past
and so it should be considered when approaching
a new problem as it solves issues
encountered in the word-based algorithm.
(page whirring)