（屏幕呼啸） (screen whooshing)
（贴纸弹出） (sticker popping)
（屏幕呼啸） (screen whooshing)
- 让我们来看看基于单词的分词。 *[译者注: token, tokenization, tokenizer 等词均译成了 分词*, 实则不翻译最佳] - Let's take a look at word-based tokenization.
基于单词的分词化的想法是 Word-based tokenization is the idea
将原始文本拆分成单词 of splitting the raw text into words
通过按空格或其他特定规则拆分， by splitting on spaces or other specific rules,
比如标点符号。 like punctuation.
在这个算法中，每个单词都有一个特定的数字 In this algorithm, each word has a specific number
或者说他的 ID。 or ID attributed to it.
在这里，"let's" 的 ID 是 250， Here, let's has the ID 250,
"do" 是 861，并且分词化 do has 861, and tokenization
后面跟感叹号的是 345。 followed by an exclamation mark has 345.
这个方法很有趣 This approach is interesting
因为模型有表示 as the model has representations
是基于整个单词的。 that are based on entire words.
单个号码所持有的信息量高， The information held in a single number is high,
因为一个词包含很多上下文 as a word contains a lot of contextual
和语义信息。 and semantic information.
然而，这种方法确实有其局限性。 However, this approach does have its limits.
比如 dog 这个词和 dogs 这个词很相似 For example, the word dog and the word dogs are very similar
他们的意思很接近。 and their meaning is close.
然而，基于单词的分词化， The word-based tokenization, however,
会给这两个词赋予完全不同的 ID will attribute entirely different IDs to these two words
因此模型将学习 and the model will therefore learn
这两个词的两个不同的嵌入。 two different embeddings for these two words.
这很不幸，因为我们想要这个模型 This is unfortunate as we would like the model
理解这些词是确实相关的， to understand that these words are indeed related,
而 dogs 只是 dog 这个词的复数形式。 and that dogs is simply the plural form of the word dog.
这种方法的另一个问题是， Another issue with this approach,
是语言中有很多不同的词。 is that there are a lot of different words in the language.
如果我们想让我们的模型理解 If we want our model to understand
该语言中所有可能的句子， all possible sentences in that language,
那么我们需要为每个不同的词设置一个 ID。 then we will need to have an ID for each different word.
以及总字数， And the total number of words,
也称为词汇量大小， which is also known as the vocabulary size,
可以很快变得非常大。 can quickly become very large.
这是一个问题，因为每个 ID 都映射到一个大向量 This is an issue because each ID is mapped to a large vector
代表这个词的意思， that represents the word's meaning,
并实现保持这些映射 and keeping track of these mappings
需要很大的模型 requires an enormous number of weights
当词汇量很大时。 when the vocabulary size is very large.
如果我们希望我们的模型保持精简， If we want our models to stay lean,
我们可以选择让分词器忽略某些词 we can opt for our tokenizer to ignore certain words
我们不一定需要的。 that we don't necessarily need.
例如，在这里，当在文本上训练我们的分词器时， For example, here, when training our tokenizer on a text,
我们可能只想使用 10,000 个最常用的单词 we might want to take only the 10,000 most frequent words
在该文本中。 in that text.
而不是从该文本中提取所有单词 Rather than taking all words from in that text
或所有语言的单词来创建我们的基本词汇。 or all languages words to create our basic vocabulary.
分词器将知道如何转换这 10,000 个单词 The tokenizer will know how to convert those 10,000 words
转换成数字，但任何其他词都会被转换 into numbers, but any other word will be converted
作为词汇外的词， to the out-of-vocabulary word,
或者像这里显示的那样，未知的词。 or like shown here, the unknown word.
不幸的是，这是一种妥协。 Unfortunately, this is a compromise.
该模型将具有完全相同的表示 The model will have the exact same representation
对于所有它不知道的单词， for all words that it doesn't know,
这可能会导致大量信息丢失 which can result in a lot of lost information
如果存在许多未知单词。 if many unknown words are present.
（屏幕呼啸） (screen whooshing)