（戏剧性的嗖嗖声） (dramatic whoosh)
- 为什么快速 tokenizer 被称 "快速"？ - Why are fast tokenizers called fast?
在这个视频中，我们将看到到底有多快， In this video, we'll see exactly how much faster,
另外，所谓的快速 tokenizer 被比较 also, so-called fast organizers are compared
和他们慢的参照物。 to their slow counterparts.
对于这个基准测试，我们将使用 GLUE MNLI 数据集 For this benchmark, we'll use the GLUE MNLI dataset
其中包含 432,000 个拼写的文本。 which contains 432,000 spells of text.
我们将看看快速和慢速版本需要多长时间 We'll see how long it takes for the fast and slow versions
一个 BERT tokenizer 来处理它们。 of a BERT tokenizer to process them all.
我们定义我们的快速和慢速的 tokenizer We define our fast and slow token organizer
使用 AutoTokenizer API。 using the AutoTokenizer API.
快速 tokenizer 在可用时是默认的。 The fast tokenizer is a default when available.
所以我们通过设置 use_fast=False 来定义成慢速。 So we pass along, use_fast=False to define the slow one.
在笔记本中，我们可以计时执行 In a notebook, we can time the execution
让其本身附带时间魔术命令，就像这样。 of itself with a time magic command, like this.
处理整个数据集快四倍 Processing the whole dataset is four times faster
使用快速 tokenizer 。 with a fast tokenizer.
确实更快，但不够吸引人。 That quicker indeed, but not very impressive.
这是因为我们传递了文本 This is because we passed along the texts
一次一个到 tokenizer 。 to the tokenizer one at a time.
这是快速 tokenizer 的常见错误 This is a common mistake to do with fast organizers
由 Rust 支持，因此能够确定优先化 which are backed by Rust, and thus able to prioritize
多个文本的 tokenization 。 the tokenization of multiple texts.
一次只向他们传递一个文本， Passing them only one text at a time,
就像在两大洲之间发送一艘货船 is like sending a cargo ship between two continents
只有一个容器，效率非常低。 with just one container, it's very inefficient.
为了释放我们快速 tokenizer 的全部速度， To unleash the full speed of our fast tokenizers,
我们需要向他们发送成批的文本，我们可以做到 we need to send them batches of texts, which we can do
使用 map 方法的 batched=True 参数。 with the batched=True argument of the map method.
现在这些都是令人印象深刻的结果，所以快速 tokenizer Now those are impressive results, so the fast tokenizer
需要 12 秒处理, 而需要 4 takes 12 second to process the dataset that takes four
分钟, 对于慢速 tokenizer 。 minute to the slow tokenizer.
结果总结于此表中， Summarizing the results in this table,
你可以看到为什么我们称 tokenizer 为快速。 you can see why we have called those tokenizers fast.
这仅用于分词化文本。 And this is only for tokenizing texts.
如果你需要训练一个新的 tokenizer ， If you ever need to train a new tokenizer,
它们也很快做到这一点。 they do this very quickly too.