（翻页） (page whirring)
- 在深入研究基于字符的分词化之前， *[译者注: token, tokenization, tokenizer 等词均译成了 分词*, 实则不翻译最佳] - Before diving in character-based tokenization,
理解为什么这种分词化很有趣 understanding why this kind of tokenization is interesting
需要了解基于单词的分词化的缺陷。 requires understanding the flaws of word-based tokenization.
如果你还没有看过第一个视频, If you haven't seen the first video
基于词的分词 on word-based tokenization
我们建议你在观看此视频之前看一下。 we recommend you check it out before looking at this video.
好的，让我们看一下基于字符的分词化。 Okay, let's take a look at character-based tokenization.
我们现在将文本拆分为单个字符， We now split our text into individual characters,
而不是文字。 rather than words.
语言中通常有很多不同的词， There are generally a lot of different words in languages,
而字符数保持较低。 while the number of characters stays low.
首先, 让我们看一下英语， To begin let's take a look at the English language,
它估计有 170,000 个不同的词， it has an estimated 170,000 different words,
所以我们需要非常大的词汇量 so we would need a very large vocabulary
来包含所有单词。 to encompass all words.
使用基于字符的词汇表， With a character-based vocabulary,
我们可以只用 256 个字符， we can get by with only 256 characters,
其中包括字母、数字和特殊字符。 which includes letters, numbers and special characters.
即使是有大量不同字符的语言 Even languages with a lot of different characters
就像中文可以有字典一样 like the Chinese languages can have dictionaries
多达 20,000 个不同的汉字 with up to 20,000 different characters
超过 375,000 个不同的词语。 but more than 375,000 different words.
所以基于字符的词汇 So character-based vocabularies
让我们使用更少的不同分词 let us use fewer different tokens
比基于单词的分词词典 than the word-based tokenization dictionaries
否则我们会使用。 we would otherwise use.
这些词汇也更全面 These vocabularies are also more complete
相较于其基于单词的词汇。 than their word-based vocabularies counterparts.
由于我们的词汇表包含所有字符 As our vocabulary contains all characters
在一种语言中的，甚至是看不见的词 used in a language, even words unseen
在分词器训练期间仍然可以分词， during the tokenizer training can still be tokenized,
因此溢出的分词将不那么频繁。 so out-of-vocabulary tokens will be less frequent.
这包括正确分词化 This includes the ability to correctly tokenize
拼错的单词，而不是丢弃它们 misspelled words, rather than discarding them
作为未知的。 as unknown straight away.
然而，这个算法也不完美。 However, this algorithm isn't perfect either.
直觉上，字符不成立 Intuitively, characters do not hold
一个词所能包含的信息量。 as much information individually as a word would hold.
例如，“让我们” 包含更多信息 For example, "Let's" holds more information
比它的第一个字母 “l”。 than it's first letter "l".
当然，并非所有语言都如此， Of course, this is not true for all languages,
作为一些语言，比如基于表意文字的语言 as some languages like ideogram-based languages
有很多信息保存在单个字符中， have a lot of information held in single characters,
但对于其他像基于字母的语言， but for others like roman-based languages,
该模型必须一次性理解多个分词 the model will have to make sense of multiple tokens at a time
以获取信息 to get the information otherwise held
在一句话中。 in a single word.
这导致了基于字符的分词器的另一个问题， This leads to another issue with character-based tokenizers,
他们的序列被翻译成非常大量 their sequences are translated into very large amount
模型要处理的分词。 of tokens to be processed by the model.
这会对上下文的大小产生影响 And this can have an impact on the size of the context
该模型将装载，并会减小 the model will carry around, and will reduce the size
可以用作模型输入文本的尺寸， of the text we can use as input for our model,
这通常是有限的。 which is often limited.
这种标记化虽然存在一些问题， This tokenization, while it has some issues,
但在过去看到了一些非常好的结果 has seen some very good results in the past
所以应该被考虑 and so it should be considered
当碰到新问题时, 来解决问题. when approaching a new problem as it solves issues
在基于词的算法中遇到的。 encountered in the word-based algorithm.
（翻页） (page whirring)