(screen whooshing)
(sticker popping)
(screen whooshing)
- Let's take a look at word-based tokenization.
Word-based tokenization is the idea
of splitting the raw text into words
by splitting on spaces or other specific rules,
like punctuation.
In this algorithm, each word has a specific number
or ID attributed to it.
Here, let's has the ID 250,
do has 861, and tokenization
followed by an exclamation mark has 345.
This approach is interesting
as the model has representations
that are based on entire words.
The information held in a single number is high,
as a word contains a lot of contextual
and semantic information.
However, this approach does have its limits.
For example, the word dog and the word dogs are very similar
and their meaning is close.
The word-based tokenization, however,
will attribute entirely different IDs to these two words
and the model will therefore learn
two different embeddings for these two words.
This is unfortunate as we would like the model
to understand that these words are indeed related,
and that dogs is simply the plural form of the word dog.
Another issue with this approach,
is that there are a lot of different words in the language.
If we want our model to understand
all possible sentences in that language,
then we will need to have an ID for each different word.
And the total number of words,
which is also known as the vocabulary size,
can quickly become very large.
This is an issue because each ID is mapped to a large vector
that represents the word's meaning,
and keeping track of these mappings
requires an enormous number of weights
when the vocabulary size is very large.
If we want our models to stay lean,
we can opt for our tokenizer to ignore certain words
that we don't necessarily need.
For example, here, when training our tokenizer on a text,
we might want to take only the 10,000 most frequent words
in that text.
Rather than taking all words from in that text
or all languages words to create our basic vocabulary.
The tokenizer will know how to convert those 10,000 words
into numbers, but any other word will be converted
to the out-of-vocabulary word,
or like shown here, the unknown word.
Unfortunately, this is a compromise.
The model will have the exact same representation
for all words that it doesn't know,
which can result in a lot of lost information
if many unknown words are present.
(screen whooshing)