（空气呼啸） (air whooshing)
- 你代码中的一些错误非常简单。 - Some bugs in your code are very straightforward.
你尝试运行它，某处出现语法错误， You try running it, you get a syntax error somewhere,
Python 准确地告诉你在哪里，然后你修复它。 Python tells you exactly where, and you fix it.
这很棒，简单而且令人满意。 This is great, it's simple and it's satisfying.
但有时，事情会崩溃 Sometimes, though, things crash
错误是无法理解的。 and the error is impossible to understand.
由于一些原因，这种情况在机器学习中经常发生， This happens a lot in machine learning for a few reasons,
你正在处理大数据结构， you're working with big data structures,
你正在使用这些大而复杂的库 you're using these big, complex libraries
有很多活动部件， with a lot of moving parts,
而且你正在做大量的 GPU 计算， and also you're doing a lot of GPU computing,
而且一般来说调试起来要困难得多。 and that in general is much more difficult to debug.
在 Keras 中还有一个问题 In Keras there's the additional problem
你的模型通常在执行前编译， that your models are often compiled before execution,
这对性能很有帮助 which is great for performance
但这也使调试它们变得非常困难。 but it makes debugging them very difficult as well.
所以，这将是一个关于该怎么做的视频 So, this is going to be a video about what to do
当你遇到那些噩梦般的一个错误时 when you run into one of those nightmare bugs
而且你只是不知道从哪里开始修复它。 and you just have no idea where to begin with fixing it.
所以，给你一些直觉 So, to give you some intuitions for
最常见的错误 the most common things that go wrong
并导致这些奇怪的问题， and cause these weird issues,
并告诉你在哪里寻找错误的来源 and show you where to look for the sources of bugs
你遇到的，让我们使用这个示例脚本。 that you encounter, let's use this example script.
因此，我将在这里分两部分向你展示。 So, I'll show it to you here in two parts.
首先，我们做所有的导入，我们加载一个数据集， First, we do all our imports, we load a dataset,
我们创建了 tokenizer 并对数据集进行分词。 we create our tokenizer and we tokenize the dataset.
接下来，我们将数据集转换为 TensorFlow 数据集， Next, we convert our datasets to TensorFlow datasets,
这就是 tf.data.Dataset， so that's tf.data.Dataset,
这样我们就可以拟合它们， and that's so that we can run fit on them,
然后我们从预训练的 checkpoint 加载我们的模型， and then we load our model from a pretrained checkpoint,
我们对其进行编译，并将其与这些数据集相匹配。 we compile it and we fit it with those datasets.
所以，这看起来很简单， So, this seems straightforward enough,
这与我们之前在课程中所做的类似。 it's similar to what we've done in the course before.
但请注意，这是令人毛骨悚然的代码 But beware, this is spooky code
并隐藏着许多黑暗而神秘的秘密。 and hides many dark and mysterious secrets.
那么，当我们运行它时会发生什么？ So, what happens when we run it?
好吧，这不是很好。 Well, it's not great.
所以，我们收到了这个错误信息，但它是什么意思呢？ So, we get this error message, but what does it mean?
我们试图训练我们的数据，但我们没有梯度？ We tried to train on our data, but we got no gradient?
这很令人困惑，我的意思是，我们从何开始 It's pretty perplexing, I mean, how do we even begin
调试没有得到梯度？ to debug not getting a gradient?
所以，当你得到的错误没有立即提示 So, when the error you get doesn't immediately suggest
问题出在哪里，最好的解决办法 where the problem is, the best solution
通常是按顺序遍历事物， is often to walk through things in sequence,
确保在每个阶段输出看起来都是正确的， making sure at each stage that the outputs look right,
那时一切看起来都很好。 that everything looks okay at that point.
而且，当然，这意味着开始的地方 And, of course, that means the place to start
总是检查你的数据。 is always to check your data.
所以，最好的办法是确保 So, the best way to make sure
你给模型的数据是好的， that the data you're giving the model is good,
是从 tf.data.Dataset 中抓取一批 is to grab a batch from the tf.data.Dataset
你的模型正在训练， that your model is training on,
那是因为它就在最后 and that's because it's right at the end
的数据 pipeline 。 of the data pipeline.
所以这意味着如果这些输出是好的， And so that means that if those outputs are good,
你可以保证你的数据 pipeline 运行良好。 you're guaranteed that your data pipeline is working well.
所以，我们可以通过遍历数据集来做到这一点 So, we can do that by looping over the dataset
进行一次迭代然后中断， for one iteration and then breaking,
这给了我们一个分批。 and that gives us a single batch.
那么，当我们检查该批次时，我们得到了什么？ So, what do we get when we inspect that batch?
我们会看到我们没有得到任何梯度 We'll see that we're not getting any gradient
因为我们没有将标签传递给 Keras。 because we're not passing labels to Keras.
所以，我们的标签在批次中， So, our labels are in the batch,
但它们是输入字典中的键 but they're a key in the input dictionary
而且它们不像 Keras 期望的那样是一个单独的标签， and they're not a separate label as Keras expects,
所以这是你会遇到的最常见的问题之一 so this is one of the most common issues you'll encounter
使用 TensorFlow 训练 Transformers 模型时。 when training Transformers models with TensorFlow.
我们的模型都可以在内部计算损失， Our models can all compute loss internally,
但要将损失用于训练 but to use that loss for training
标签需要在输入字典中传递， the labels need to be passed in the input dictionary,
其模型可以看到它们。 where the model can see them.
这种内部损失是我们使用的损失 This internal loss is the loss that we use
当我们在调用编译时没有指定损失时， when we don't specify a loss when we call compile,
当我们不指定损失参数时。 when we don't specify a loss argument.
所以，另一方面，Keras， So, Keras, on the other hand,
通常期望标签单独传递 usually expects labels to be passed separately
从输入字典， from the input dictionary,
并且对模型不可见， and not to be visible to the model,
损失计算通常会失败 and loss computations will usually fail
如果你不那样做 if you don't do that
所以我们需要选择其中之一， So we need to choose one or the other,
要么我们使用模型的内部损失 either we use the model's internal loss
并将标签保留在原处， and keep the labels where they are,
或者我们继续使用 Keras 损失 or we keep using Keras losses
但我们将标签移动到 Keras 期望的地方。 but we move the labels to the place Keras expects them.
所以，还是简单点，让我们解决这个问题 So, or simplicity here, let's fix this issue
通过使用模型的内部损失， by using the model's internal losses,
我们通过删除损失参数来做到这一点 and we do that by removing the loss argument
从调用编译。 from the call to compile.
那么，如果我们现在尝试训练会发生什么？ So, what happens if we try training now?
所以我们用它重新编译，我们调用 model.fit，会发生什么？ So we recompile with that, we call model.fit, what happens?
好吧，这次它运行了，但现在我们消失了 NaN。 Well, it runs this time but now we get a loss of NaN.
所以，这不好，NaN 表示不是数字 So, that's not good, NaN means not a number
总的来说，这不是一个好损失。 and it's not a good loss to have in general.
事实上，如果我们现在检查我们的模型， In fact, if we inspect our model now,
我们会看到不仅所有的输出都是 NaN， we'll see that not only are all the outputs NaN,
所有的权重和损失也是 NaN。 all the weights are NaN as well, as well as the loss.
所以一旦一个 NaN 悄悄爬进你的计算， So once a single NaN creeps into your computations,
它倾向于传播，因为它从损失传播 it tends to spread, because it propagates from the loss
一旦它处于损失状态，它就会处于梯度状态， and once it's at the loss it's at the gradient,
它达到梯度， it gets to the gradient,
然后一旦它处于梯度中 and then once it's in the gradient
它进入权重更新， it enters the weight updates,
然后你所有的权重更新最终也都是 NaN 。 and then all your weight updates end up as NaN as well.
所以 NaN 在这里完全破坏了我们的模型， So NaN just completely destroyed our model here,
但它首先潜入何处？ but where did it creep in first?
所以要找出答案，我们需要回到一个点 So to find out, we need to back to a point
在模型被摧毁之前， before the model was destroyed,
我们需要重新初始化模型 we need to re-initialize the model
并查看第一批的输出。 and look at the outputs for just the first batch.
当我们这样做时， And when we do that,
我们看到 NaN 首先出现在损失中， we see that NaN first appears in the loss,
但仅在某些样本中。 but only in some samples.
所以你可以更详细地看到这个 So you can see this in more detail
在课程笔记的随附部分中， in the accompanying section of the course notes,
我在这里移动得相当快， I am moving fairly quickly here,
但我们发现如果我们看一下标签， but we find that if we look at the labels,
损失为 NaN 的样本的标签均为 2。 the samples with a loss of NaN all have a label of two.
所以这给了我们一个非常有力的线索， So this gives us a very strong clue,
如果我们使用 model.config.num_labels 检查模型， if we check the model with model.config.num_labels,
我们看到模型认为只有两个标签， we see that the model thinks there's only two labels,
但如果我们看到值为二， but if we see a value of two,
这意味着至少有三个标签 that means there's at least three labels
因为 0 也是一个标签。 because 0 is a label as well.
所以我们得到大量 NaN So we got a lots of NaN
因为我们的标签集中有一个 “不可能” 的标签， because we got an "impossible" label in our label set,
并修复我们需要返回并设置模型 and to fix that we need to go back and set the model
期待正确数量的标签， to expect the right number of labels,
所以我们可以设置 num_labels=3 so we can set num_labels=3
当我们初始化模型用 from_pretrained 时， when we initialize the model but from_pretrained,
现在希望我们可以避免这个问题。 and now hopefully we can avoid this issue.
所以，现在我们认为我们的数据很好，我们的模型也很好 So, now we think our data is good and our model is good
所以训练应该有效 and so training should work
但是如果我们尝试运行 model.fit，我们，嗯…… but if we try running model.fit, we, well...
我的意思是，我们确实有损失， I mean, we do get a loss,
这是一个数字，它正在下降 it is a number and it is going down
但它不会很快下降 but it's not going down very quickly
如果我们一直运行这个， and if we keep running this out,
我们会发现它停在相当高的损失值处。 we'll find that it stalls at a fairly high loss value.
发生什么了？ So, what's going on?
好吧，当一切正常时， Well, when things are mostly working,
但训练只是很慢或有点奇怪， but training is just slow or a bit odd,
这通常是查看优化器的好时机 that can often be a good time to look at your optimizer
和你的训练超参数。 and your training hyperparameters.
这就是我想提的地方 And this is where I want to mention
最常见的问题来源之一 one of the most common sources of issues
当你使用 Keras 时， when you're working with Keras,
你可以用字符串命名优化器， you can name things like optimizers with strings,
所以 Keras 支持它，而且非常方便， so Keras supports that and it's very convenient,
但如果你这样做所有的选择 but if you do that all of the options
默默地设置为默认值。 get silently set to their default values.
所以我们将优化器指定为 Adam， So we specified our optimizer as Adam,
但在这个过程中我们无形中得到了 but in the process we invisibly got
默认学习率，即 1e-3， the default learning rate, which is 1e-3,
或 10 的 -3 次方。 or 10 to the power of -3.
所以这个学习率太高了 So this learning rate is way too high
用于训练 transformer 模型， for training transformer models,
我们应该回去直接指定学习率， we should go back and specify the learning rate directly,
不使用字符串。 not using a string.
所以，这里的好值在 1e-5 和 1e-4 之间 So, good values here are between 1e-5 and 1e-4
所以让我们平分差价并选择 5e-5。 so let's split the difference and pick 5e-5.
所以如果你用那个重新编译， So if you recompile with that,
你最终会发现训练确实有效。 you'll find that training actually works, at last.
损失有效减少 The loss goes down efficiently
并且收敛到一个较低的值。 and it converges to a lower value.
所以，再一次，我确实很快地完成了这个 So, again, I did go through this quite quickly
我强烈建议查看课程笔记 and I strongly recommend checking out the course notes
要更详细地了解这一点， to see this in more detail,
并自己试验代码 and to experiment with the code yourself
看看错误是什么样的 and see what the errors look like
以及如何解决他们， and how you can approach them,
但我希望我在这里给了你一个简短的总结 but I hope I've given you here a quick summary
最常见的错误 of the most common bugs
也许是最常见的调试方法 and maybe the most common debugging approaches
来对付他们。 to dealing with them.
所以，祝你好运，记得多休息 So, good luck, and remember to take plenty of breaks
如果你的代码给你带来困难。 if your code is giving you a hard time.
（空气呼啸） (air whooshing)