（画面沙沙作响） (screen swishing)
（文字嗖嗖） (text swishing)
在我们的其他视频中， In our other videos,
我们讨论了基础知识 we talked about the basics
使用 Tensorflow 微调语言模型， of fine-tuning a language model with Tensorflow,
和往常一样，当我提到视频时，我会在下面链接它们。 and as always, when I refer to videos I'll link them below.
不过，我们可以做得更好吗？ Still, can we do better?
这是我们模型微调视频中的代码， So here's the code from our model fine-tuning video,
在它起作用的同时， and while it works,
我们绝对可以调整一些东西。 we could definitely tweak a couple of things.
到目前为止，最重要的是学习率。 By far the most important thing is the learning rate.
在本视频中，我们将讨论如何更改它， In this video we'll talk about how to change it,
这将使你的训练 which will make your training
更加稳定地成功。 much more consistently successful.
其实真的有两件事 In fact, really there are two things
我们想改变 Adam 的默认学习率。 we want to change about the default learning rate for Adam.
所以首先我们要改变 So the first we want to change
是它对我们的模型来说太高了， is that it's way too high for our models,
默认情况下，Adam 使用学习率 by default, Adam uses a learning rate
10 的 -3 次方，1e-3， of 10 to the minus 3, 1e-3,
这对于训练 transformer 模型来说非常高。 and that's very high for training transformer models.
我们将从 5 乘 10 的负 5次方 开始， We're going to start at 5 by 10 to the minus 5,
5e-5，比默认值低 20 倍。 5e-5, which is 20 times lower than the default.
其次，我们不只是想要一个恒定的学习率， And secondly, we don't just want a constant learning rate,
我们可以获得更好的性能 we can get even better performance
如果我们将学习率降低到一个很小的值， if we decay the learning rate down to a tiny value,
甚至在训练过程中为零。 or even to zero , over the course of training.
这就是这里的东西， So that's what this thing here,
这个 Polynomial Decay schedule 事情正在做。 this Polynomial Decay schedule thing is doing.
等会儿我会告诉你衰减是什么样子的， So I'll show you what that decay looks like in a second,
但首先我们需要告诉规划程序 but first we need to tell the scheduler
训练将持续多长时间， how long training is going to be,
以便它以正确的速度衰减， so that it decays at the right speed,
这就是这里的代码所做的。 and that's what this code here is doing.
我们正在计算有多少小批量 We're computing how many minibatches
该模型将在其整个训练过程中进行观察， the model is going to see over its entire training run,
这是训练集的大小， which is the size of the training set,
然后我们乘以它 and then we multiply that
按 epoch 数 by the number of epochs
获得 batch 总数 to get the total number of batches
在整个训练过程中。 across the whole training run.
一旦我们知道我们进行了多少训练步骤， Once we know how many training steps we're taking,
我们只是将所有这些信息传递给规划程序 we just pass all that information to the scheduler
我们准备好了。 and we're ready to go.
多项式衰减时间表是什么样的？ What does the polynomial decay schedule look like?
嗯，看起来像这样， Well, it looks like this,
它从 5e-5 开始， it starts at 5e-5,
这意味着 5 乘以 10^{-5}， which means 5 times 10 to the minus 5,
然后以恒定速率衰减 and then decays down at a constant rate
直到它在训练结束时达到零。 until it hits zero right at the very end of training.
所以等一下，我已经能听到你的声音了 So hold on, I can already hear you
不过，对着你的显示器大喊大叫， yelling at your monitor, though,
是的，我知道， and yes, I know,
这实际上是常数或线性衰减， this is actually constant or a linear decay,
我知道这个名字有 "多项式"， and I know the name is polynomial,
你感觉被骗了，你知道的， and you're feeling cheated that, you know,
你被许诺了一个多项式但还没有得到它， you were promised a polynomial and haven't gotten it,
所以冷静下来，没关系， so calm down though, it's okay,
因为，当然，线性函数只是 because, of course, linear functions are just
一阶特例 first-order special cases
的一般多项式函数， of the general polynomial functions,
如果你调整这个类的选项， and if you tweak the options to this class,
你可以获得一个真正的多项式， you can get a truly polynomial,
高阶衰减时间表， a higher-order decay schedule,
但现在这个线性时间表对我们来说还行， but this linear schedule will work fine for us for now,
我们实际上并不需要所有这些 we don't actually need all those
花哨的调整和花哨的小工具。 fancy tweaks and fancy gadgets.
所以回来， So coming back,
我们如何实际使用这个学习率计划 how do we actually use this learning rate schedule
一旦我们创造了它？ once we've created it?
所以很简单，我们只需将其传递给 Adam。 So it's simple, we just pass it to Adam.
所以我们第一次编译模型时， So the first time we compiled the model,
我们刚刚传递了字符串 Adam， we just passed the string Adam,
得到我们的优化器。 to get our optimizer.
所以 Keras 识别常见优化器的名称 So Keras recognizes the names of common optimizers
和损失函数，如果你将它们作为字符串传递， and loss functions if you pass them as strings,
这样可以节省时间 so it saves time to do that
如果你只想要默认设置。 if you only want the default settings.
但现在我们是专业的机器学习者， But now we're professional machine learners,
而且，你知道，薪资审查即将到来， and, you know, that salary review is upcoming,
所以我们有自己的学习率时间表， so we've got our very own learning rate schedule,
我们会把事情做好。 and we're gonna do things properly.
所以我们首先要做的是导入优化器， So the first we do is we import the optimizer,
然后我们用调度程序初始化它， and then we initialize it with a scheduler,
它被传递给学习率参数 which is getting passed to to the learning rate argument
对该优化器的。 of that optimizer.
现在我们用这个新的优化器编译模型， And now we compile the model with this new optimizer,
再一次，无论你想要什么损失函数， and again, whatever loss function you want,
所以这将是稀疏的分类交叉熵 so this is going to be sparse categorical crossentropy
如果你正在关注 fine-tuning (微调) 的视频。 if you're following along from the fine-tuning video.
然后，我们准备好了， And then, we're we're ready to go,
现在我们有了一个高性能模型， now we have a high-performance model,
并准备接受训练。 and ready for training.
剩下的就是拟合模型 All that remains is to fit the model
就像我们以前做的一样。 just like we did before.
记住，因为我们编译了模型 Remember, because we compiled the model
使用新的优化器和新的学习率规划， with the new optimizer and the new learning rate schedule,
我们实际上根本不需要改变任何东西 we actually don't need to change anything at all
当我们调用 fit 时， when we call fit,
我们只是再次调用它， we just call it again,
使用与之前完全相同的命令， with exactly the same command as before,
但现在我们得到了很好的训练， but now we get a beautiful training,
有一个很好的，平滑的学习率衰减， with a nice, smooth learning rate decay,
从好的值开始， starting from a good value,
并衰减到零。 and decaying down to zero.
（画面沙沙作响） (screen swishing)