- 让我们来看看基于子词的分词。 *[译者注: token, tokenization, tokenizer 等词均译成了 分词*, 实则不翻译最佳] - Let's take a look at subword based tokenization.
了解为什么基于子词的分词是 Understanding why subword based tokenization
是有趣的需要理解 is interesting requires understanding the flaws
基于单词和基于校正器分词化的缺陷。 of word based and corrector based tokenization.
如果你还没有看过第一个视频 If you haven't seen the first videos
关于基于单词和基于字符的标记化 on word based and character based tokenization
我们建议你观看它们 we recommend you check them out
在看这个视频之前。 before looking at this video.
基于子词的分词化介于基于字符 Subword based tokenization lies in between character based
和基于单词的分词算法之间。 and word based tokenization algorithms.
这个想法是找到一个中间场 The idea is to find a middle ground
在很大的词汇表, between very large vocabularies,
大量的词汇分词 a large quantity of out vocabulary tokens
还有在非常相似的词之间意义差 and a loss of meaning across very similar words
对基于单词的分词器和非常长的序列 for word based tokenizers and very long sequences
以及意义不大的单个标记 as well as less meaningful individual tokens
对于基于字符的分词器之间。 for character based tokenizers.
这些算法依赖于以下原则。 These algorithms rely on the following principle.
常用词不宜拆分 Frequently used words should not be split
成更小的子词 into smaller subwords
而稀有词应该被分解 while rare words should be decomposed
成有意义的子词。 into meaningful subwords.
一个例子是 dog 这个词。 An example is the word dog.
我们想让我们的分词器有一个单一的 ID We would like to have our tokenizer to have a single ID
对于 dog 这个词 for the word dog rather
而不是将其拆分为字母 d o 和 g。 than splitting it into characters d o and g.
然而，当遇到 dog 这个词时 However, when encountering the word dogs
我们希望我们的分词从词根上理解这一点 we would like our tokenize to understand that at the root
这还是 dog 这个词。 this is still the word dog.
添加 s 后，意思略有改变 With an added "s", that slightly changes the meaning
同时保持最初的意思。 while keeping the original idea.
另一个例子是像 tokenization 这样的复杂词 Another example is a complex word like tokenization
可以拆分成有意义的子词。 which can be split into meaningful subwords.
这个词的根 The root of the word
是 token ，以及 -ization 完整了这个词 is token and -ization completes the root
以赋予它稍微不同的含义。 to give it a slightly different meaning.
拆分这个词是有道理的 It makes sense to split the word
一分为二，token 作为词根， into two, token as the root of the word,
标记为单词的开头 labeled as the start of the word
和 -ization 作为标记的附加信息 and -ization as additional information labeled
作为单词的完整化。 as a completion of the word.
如此一来，该模型现在将能够有作用 In turn, the model will now be able to make sense
在不同情况下的分词。 of token in different situations.
它会理解这个词的形式: token, tokens, tokenizing It will understand that the word's token, tokens, tokenizing
和 tokenization 具有相似的含义并且是相关联的。 and tokenization have a similar meaning and are linked.
它还将理解 tokenization 、modernization It's will also understand that tokenization, modernization
和 immunization ，都有相同的后缀 and immunization, which all have the same suffixes
可能在相同的句法情况下使用。 are probably used in the same syntactic situations.
基于子词的分词器通常有办法来 Subword based tokenizers generally have a way to
识别哪些分词是单词的开头 identify which tokens are a start of word
以及哪些分词完成了单词的开始。 and which tokens complete start of words.
所以这里以分词作为单词的开始 So here token as the start of a word
以及 ##ization 的开始作为单词的完成。 and ##ization as completion of a word.
这里 ## 表示 -ization 是单词的一部分 Here, the ## prefix indicates that ization is part of a word
而不是它的开始。 rather than the beginning of it.
## 记号 The ## comes
来自基于单词片算法的 BERT 分词器。 from the BERT tokenizer based on the word piece algorithm.
其他分词器使用其他前缀, 可以是 Other tokenizes use other prefixes which can be
用来表示单词的一部分 placed to indicate part of words
比如在这里或单词的开头。 like in here or start of words instead.
有很多 There are a lot
可以使用的不同算法 of different algorithms that can be used
用于子词分词化 for subword tokenization
大多数模型都获得了目前最先进的结果 and most models obtaining state-of-the-art results
今日英语 in English today
使用一些子词标记化算法。 use some kind of subword tokenization algorithms.
这些方法有助于减少词汇量 These approaches help in reducing the vocabulary sizes
通过不同词之间共享的信息 by sharing information across different words
有能力以理解前缀 having the ability to have prefixes
和后缀如斯。 and suffixes understood as such.
他们在非常相似的词中保留意义 They keep meaning across very similar words
通过识别相似的分词，将它们组合起来。 by recognizing similar tokens, making them up.