(intro whooshing)
(smiley snapping)
(words whooshing)
- In the next few videos,
we'll take a look at the tokenizers.
In natural language processing,
most of the data that we handle consists of raw text.
However, machine learning models
cannot read or understand text in its raw form,
they can only work with numbers.
So the tokenizer's objective
will be to translate the text into numbers.
There are several possible approaches to this conversion,
and the objective
is to find the most meaningful representation.
We'll take a look at three distinct tokenization algorithms.
We compare them one to one,
so we recommend you take a look at the videos
in the following order.
First, "Word-based,"
followed by "Character-based,"
and finally, "Subword-based."
(outro whooshing)