- 让我们谈谈 transformer 的碳足迹。 - So let's talk about the carbon footprint of transformers.
也许您看过这样的头条新闻 Maybe you've seen headlines such as this one
训练单个 AI 模型排放的碳含量 that training a single AI model can emit as much carbon
相当于五辆汽车生命周期的总排放量。 as five cars in their lifetimes.
那么什么时候是真的，而且总是真的吗？ So when is this true and is it always true?
其实呢，这实际上取决于几个因素。 Well, it actually depends on several things.
最重要的一点，这取决于 Most importantly, it depends
您所使用的能源类型。 on the type of energy you're using.
如果您使用的是可再生能源，例如 If you're using renewable energy such as
太阳能、风能、水力发电，那么 solar, wind, hydroelectricity, you're really
根本不会排放任何碳，非常非常少。 not emitting any carbon at all, very, very little.
如果您使用的是煤炭等不可再生能源 If you're using non-renewable energy sources such as coal
那么它们的碳足迹要高得多 then their carbon footprint is a lot higher
因为本质上您正在排放大量的温室气体。 'cuz essentially you are emitting a lot of greenhouse gases.
另一个因素是训练时间。 Another aspect is training time.
所以您训练的时间越长，消耗的能量就越多 So the longer you train, the more energy you use
您使用的能源越多，您排放的碳就越多，对吗？ the more energy you use, the more carbon you emit, right?
所以所有这些因素一起进行考虑 So this really adds up
特别是如果您正在训练大型模型 especially if you're training large models for
且持续了数小时、数天或数周的时间。 for hours and days and weeks.
您使用的硬件也很重要 The hardware you use also matters
例如某些 GPU 效率更高 because some GPUs, for example, are more efficient
相比较别的硬件来说，其利用效率使用得当。 than others and utilizing efficiency use properly.
一直能够百分百地被使用 So using them a hundred percent all the time
可以真正减少您的能源消耗。 can really reduce the energy consumption that you have.
进一步减少您的碳足迹。 And then once again, reduce your carbon footprint.
还有其他因素比如 IO There's also other aspects such as IO
比如数据，等等。 such as data, et cetera, et cetera.
但这三点是您应该关注的主要因素。 But these are the main three that you should focus on.
所以当我谈论能源和碳强度时 So when I talk about energy sources and carbon intensity
那个的真实意义是什么？ what does that really mean?
所以如果您看屏幕顶部 So if you look at the top of the screen
您可以看到印度孟买的云计算实例 you have a carbon footprint
所产生的碳足迹 of a cloud computing instance in Mumbai, India
每千瓦时排放 920 克二氧化碳。 which emits 920 grams of CO2 per kilowatt hour.
这差不多有一公斤 This is almost one kilogram
每千瓦时电力使用的二氧化碳排放量。 of CO2 per kilowatt hour of electricity used.
如果您把它与加拿大蒙特利尔， If you compare that with Canada, Montreal
也就是我现在所在的位置相比，每千克小时排放 20 克二氧化碳。 where I am right now, 20 grams of CO2 per kilo hour.
所以它们有着非常大的区别。 So that's a really, really big difference.
碳排放量几乎增加了 40 倍 Almost more than 40 times more carbon emitted
在孟买对蒙特利尔。 in Mumbai versus Montreal.
所以这真的都需要考虑进去。 And so this can really, really add up.
例如，如果您要训练一个模型数周 If you're training a model for weeks, for example
您乘以 40 you're multiplying times 40
您排放的碳。 the carbon that you're emitting.
因此选择合适的实例 So choosing the right instance
选择低碳计算实例 choosing a low carbon compute instance
这真的是您能做的最有影响力的事情。 is really the most impactful thing that you can do.
这就是它真正可以产生影响的地方 And this is where it can really add up
如果您正在一个碳密集的地区 if you're training in a very intensive
进行非常密集的训练 in a very carbon intensive region
其他要考虑的因素，例如 other elements to consider, for example
使用预训练模型 using pre-trained models
这就是回收的机器学习等价物。 that's the machine learning equivalent of recycling.
当您有可用的预训练模型时 When you have pre-trained models available using them
您根本没有排放任何碳，对吧？ you're not emitting any carbon at all, right?
因为您没有在训练任何东西。 You're not retraining anything.
因此先看看当前已经有了哪些工具 So that's also doing your homework
能够帮助您处理所需要进行的任务。 and kind of looking around what already exists.
微调而不是从头开始训练。 Fine tuning instead of training from scratch.
所以再一次 So once again
如果您找到几乎是您需要的模型 if you find a model that is almost what you need
但对最后几层的调整不是很精细 but not quite fine tuning the last couple of layers
通过调整来达到目的 making it really fit your purpose instead
而不是从头通过训练 transformer of training a large transformer
这样的话会大大提高您的效率 from scratch can really help,
从较小的实验开始 starting with smaller experiments
并边调试边工作。 and debugging as you go.
这意味着，例如，培训 So that means, for example, training
弄清楚数据编码 figuring out data encoding, figuring out, you know
确保没有小错误 making sure that there's no small bugs, that you'll
您会意识到，经过 16 个小时的训练 you'll realize, you know, 16 hours into training
从小事做起，真正确保 starting small and really making sure
您在做什么，您的代码是什么，这样才是稳妥的。 that what you're doing, what your code is, is stable.
最后做一个文献综述 And then finally doing a literature review to
选择超参数范围，然后跟随 choose hyper parameter ranges and then following
使用随机搜索而不是网格搜索。 up with a random search instead of a grid search.
所以随机搜索超参数 So random searches for hyper parameters
在寻找最佳配置作为网格搜索时 combinations have actually shown to be as efficient
组合实际上被证明是有效的。 in finding the optimal configuration as grid search.
但显然您并没有尝试所有可能的组合 But obviously you're not trying all possible combinations
您只是在尝试其中的一部分。 you're only trying a subset of them.
所以这也很有帮助。 So this can really help as well.
所以现在如果我们回去 So now if we go back
2019 年 Strubell 等人的原始论文 to the original paper by Strubell et all in 2019
关于那五辆车的论文中。 the infamous five cars in their lifetimes paper.
如果您只是考虑 If you just look
一个 2 亿周边 transformer 的因素 at a transformer of 200 million perimeter transformer
它的碳足迹约为 200 磅二氧化碳 it is carbon footprint is around 200 pounds of CO2
这很多 which is significant
但它离五辆汽车还差得很远，对吧？ but it's nowhere near five cars, right?
这甚至不是跨大西洋航班。 It's not even a transatlantic flight.
它真正到达这一量级的方式是当您在做 How it really adds up is when you're doing
神经架构搜索的时候 neural architecture search
当您进行超参数调整时，以及 when you're doing hyper parameter tuning, and
在尝试所有可能的组合的时候 this is trying all possible combinations
等等，等等。 et cetera, et cetera.
这是就像 And this is where
就像 600,000 磅的二氧化碳来自哪里一样。 like the 600,000 pounds of CO2 came from.
所以这真的是需要把所有因素考虑进来才可以。 So this is really where things add up.
所以，但如果您正认真地做事 So, but if you're doing things mindfully and conscientiously
那么您的碳足迹就不会那么大， then your carbon footprint wont be as big as,
正如本文所暗示的，一些工具可以用来计算 as the paper implied, some tools to figure
出您排放的 CO2 量。 out how much CO2 exactly you're emitting.
有一个基于网络的工具叫做 machine There's a web-based tool called machine
学习提交计算器，它可以让您 learning submissions calculator, which allows you
手动输入，例如，您使用的硬件 to manually input, for example, which hardware you used
您用了多少小时 how many hours you used it for
它位于本地或云端。 where it was located locally or in the cloud.
然后它会给您一个估计 And then it's gonna give you an estimate
您排放了多少二氧化碳。 of how much CO2 you emitted.
另一个以编程方式执行此操作的工具， Another tool that does this programmatically,
称为代号碳。 is called code carbon.
您可以 PIP 安装它，您可以去 GitHub So you can PIP install it, you can, you can go to the GitHub
它与您的代码并行运行。 and essentially it runs in parallel to your code.
所以基本上您调用它 So essentially you call it
然后交给它做所有的训练。 and then you do all your training.
最后它会给您一个估计 And then at the end it's gonna give you an estimate
包含排放量估算值的 CSV 文件。 a CSV file with an estimate of your emissions.
它会给您一些比较。 And it's gonna give you some comparisons.
它有一个可视化用户界面，您可以在其中真正看到 It's got a visual UI where you can really look
这与开车或看电视相比如何。 at how this compares to driving a car or watching TV.
所以它可以给您一个想法 So it can give you an idea
您的排放范围也是如此。 of the scope of your emissions as well.
实际上，code carbon 已经集成到 auto And actually, code carbon is already integrated into auto
和 LP 中，希望人们能够 and LP and hopefully people will be using it
开箱即用，轻松跟踪所有训练和部署 transformer out of the box and easily tracking their emissions all
的碳排放。 through training and deploying transformers.