（微妙的爆炸） (subtle blast)
- 在这个视频中，我们将一起看到 - In this video, we will see together
什么是规范化组件 what is the normalizer component
我们会在每个 tokenizer 的开头找到它。 that we'd find at the beginning of each tokenizer.
规范化操作包括 The normalization operation consists
在应用一系列规范化规则 in applying a succession of normalization rules
到原始文本时。 to the raw text.
我们选择规范化规则来去除文本中的噪音 We choose normalization rules to remove noise in the text
这似乎对学习没有用 which seem useless for the learning
以及使用我们的语言模型。 and use of our language model.
让我们来看一个非常多样化的句子，用不同的字体 Let's take a very diverse sentence with different fonts,
大写和小写的字符， upper and lower case characters,
重音符号、标点符号和多个空格， accents, punctuation and multiple spaces,
看看几个 tokenizer 是如何规范化它的。 to see how several tokenizer normalize it.
来自 FNet 模型的 tokenizer The tokenizer from the FNet model
用字体变体改变了字母 has transformed the letter with font variants
或圈入他们的基本版本 or circled into their basic version
并删除了多个空格。 and has removed the multiple spaces.
现在，如果我们看一下规范化 And now if we look at the normalization
使用 Retribert 的分词器， with Retribert's tokenizer,
我们可以看到它保留了字符 we can see that it keeps characters
具有多种字体变体并保留多个空格， with several font variants and keeps the multiple spaces,
但它删除了所有的重音。 but it removes all the accents.
如果我们继续测试这种标准化 And if we continue to test this normalization
与模型相关的许多其他 tokenizer of many other tokenizers associated to models
我们可以在 Hub 上找到， that we can find on the Hub,
我们看到他们还提出了其他类型的正常化。 we see that they also propose other kind of normalization.
使用快速 tokenizer ， With the fast tokenizers,
很容易观察到选择的规范化 it's very easy to observe the normalization chosen
对于当前加载的分词器。 for the currently loaded tokenizer.
事实上，每个快速 tokenizer 的实例 Indeed, each instance of a fast tokenizer
有一个底层 tokenizer has an underlying tokenizer
来自存储的 HuggingFace Tokenizers 库 from the HuggingFace Tokenizers library stored
在 backend_tokenizer 属性中。 in the backend_tokenizer attribute.
这个对象本身有一个规范器属性 This object has itself a normalizer attribute
我们可以使用, 多亏 normalize_str 方法 that we can use thanks to the normalize_str method
以规范化一个字符串。 to normalize a string.
因此，这种标准化非常实用， It is thus very practical that this normalization,
使用在训练时 which was used at the time of the training
保存分词器， of the tokenizer was saved,
并且它会自动应用 and that it applies automatically
当你要求训练过的 tokenizer 对文本进行分词时。 when you ask a trained tokenizer to tokenize a text.
例如，如果我们没有包含 albert 标准化器， For example, if we hadn't included the albert normalizer,
我们会有很多未知的 token we would have had a lot of unknown tokens
通过标记这个句子 by tokenizing this sentence
带有重音符号和大写字母。 with accents and capital letters.
这种转变也可能是无法检测的 This transformation can also be undetectable
通过简单的打印出来。 with a simple print.
确实，请记住，对于计算机来说， Indeed, keep in mind that for a computer,
文本只是连续的 0 和 1， text is only a succession of 0 and 1,
碰巧是不同的承接 and it happens that different successions
让 0 和 1 呈现相同的打印字符。 of 0 and 1 render the same printed character.
0 和 1 以 8 个为一组组成一个字节。 The 0 and 1 go in group of 8 to form a byte.
然后计算机必须解码这个字节序列 The computer must then decode this sequence of bytes
成一系列代码点。 into a sequence of code points.
在我们的示例中，2 个字节通过 UTF-8 解码 In our example, the 2 bytes is decoded using UTF-8
成一个单一的代码点。 into a single code point.
然后 unicode 标准允许我们 The unicode standard then allows us
找到与此代码点对应的字符， to find the character corresponding to this code point,
c 音符。 the c cedilla.
让我们重复同样的操作 Let's repeat the same operation
有了这个由 3 个字节组成的新序列， with this new sequence composed of 3 bytes,.
这次转化为两个码点， This time it is transformed into two code points,
这也对应于 c cedilla 字符。 which also correspond to the c cedilla character.
它实际上是组成 It is in fact the composition
unicode 拉丁文小写字母 C of the unicode Latin Small Letter C
和组合的音符。 and the combining cedilla.
但这很烦人，因为在我们看来 But it's annoying because what appears to us
为了成为一个单一的字符 to be a single character
对于计算机来说完全不是一回事。 is not at all the same thing for the computer.
幸好有 unicode 标准化标准 Fortunately, there are unicode standardization standards
称为 NFC、NFD、NFKC 或 NFKD known as NFC, NFD, NFKC or NFKD
这允许消除其中的一些差异。 that allow erasing some of these differences.
这些标准通常由 tokenizer 使用。 These standards are often used by tokenizers.
在所有这些前面的例子中， On all these previous examples,
即使规范化改变了文本的外观， even if the normalizations changed the look of the text,
他们没有改变内容； they did not change the content;
你仍然可以阅读， you could still read,
“Hello world，让我们规范一下这句话。” "Hello world, let's normalize this sentence."
但是，你必须知道一些规范化 However, you must be aware that some normalizations
如果他们不适配他们的语料库，可能会非常有害。 can be very harmful if they are not adapted to their corpus.
例如，如果你使用法语句子， For example, if you take the French sentence,
“Un pere indigne”，意思是 “愤怒的父亲”， "Un pere indigne," which means "An indignant father,"
并使用 bert-base-uncase tokenizer 对其进行规范化 and normalize it with the bert-base-uncase tokenizer
这消除了口音， which removes the accent,
然后句子变成 “Un pere indigne” then the sentence becomes "Un pere indigne"
意思是 “一个不称职的父亲”。 which means "An unworthy father".
如果你观看此视频是为了构建自己的 tokenizer ， If you watched this video to build your own tokenizer,
没有绝对的规则选择与否 there are no absolute rules to choose or not
一个新 tokenizer 的规范化， a normalization for a new tokenizer,
但我建议你花时间选择它们 but I advise you to take the time to select them
这样它们就不会让你丢失重要信息。 so that they do not make you lose important information.
（微妙的爆炸） (subtle blast)