（介绍呼呼） (intro whooshing)
（笑脸拍打） (smiley snapping)
（话语嘶哑） (words whooshing)
- 在接下来的几个视频中， - In the next few videos,
我们将看一下分词器 *[译者注: token, tokenization, tokenizer 等词均译成了 分词*, 实则不翻译最佳] we'll take a look at the tokenizers.
在自然语言处理中， In natural language processing,
我们处理的大部分数据都是原始文本。 most of the data that we handle consists of raw text.
然而，机器学习模型 However, machine learning models
无法阅读或理解原始形式的文本， cannot read or understand text in its raw form,
他们只能理解数字。 they can only work with numbers.
所以分词器的目标 So the tokenizer's objective
将是把文本翻译成数字。 will be to translate the text into numbers.
这种转换有几种可能的方法， There are several possible approaches to this conversion,
并且目标 and the objective
是找到最有意义的表示。 is to find the most meaningful representation.
我们将看看三种不同的分词化算法。 We'll take a look at three distinct tokenization algorithms.
我们对其一对一比较， We compare them one to one,
所以我们建议你看一下视频 so we recommend you take a look at the videos
按以下顺序。 in the following order.
首先，“基于单词”， First, "Word-based,"
其次是 “基于字符”， followed by "Character-based,"
最后，“基于子词”。 and finally, "Subword-based."
（结尾嘶嘶声） (outro whooshing)