- So let's talk about the carbon footprint of transformers.
Maybe you've seen headlines such as this one
that training a single AI model can emit as much carbon
as five cars in their lifetimes.
So when is this true and is it always true?
Well, it actually depends on several things.
Most importantly, it depends
on the type of energy you're using.
If you're using renewable energy such as
solar, wind, hydroelectricity, you're really
not emitting any carbon at all, very, very little.
If you're using non-renewable energy sources such as coal
then their carbon footprint is a lot higher
'cuz essentially you are emitting a lot of greenhouse gases.
Another aspect is training time.
So the longer you train, the more energy you use
the more energy you use, the more carbon you emit, right?
So this really adds up
especially if you're training large models for
for hours and days and weeks.
The hardware you use also matters
because some GPUs, for example, are more efficient
than others and utilizing efficiency use properly.
So using them a hundred percent all the time
can really reduce the energy consumption that you have.
And then once again, reduce your carbon footprint.
There's also other aspects such as IO
such as data, et cetera, et cetera.
But these are the main three that you should focus on.
So when I talk about energy sources and carbon intensity
what does that really mean?
So if you look at the top of the screen
you have a carbon footprint
of a cloud computing instance in Mumbai, India
which emits 920 grams of CO2 per kilowatt hour.
This is almost one kilogram
of CO2 per kilowatt hour of electricity used.
If you compare that with Canada, Montreal
where I am right now, 20 grams of CO2 per kilo hour.
So that's a really, really big difference.
Almost more than 40 times more carbon emitted
in Mumbai versus Montreal.
And so this can really, really add up.
If you're training a model for weeks, for example
you're multiplying times 40
the carbon that you're emitting.
So choosing the right instance
choosing a low carbon compute instance
is really the most impactful thing that you can do.
And this is where it can really add up
if you're training in a very intensive
in a very carbon intensive region
other elements to consider, for example
using pre-trained models
that's the machine learning equivalent of recycling.
When you have pre-trained models available using them
you're not emitting any carbon at all, right?
You're not retraining anything.
So that's also doing your homework
and kind of looking around what already exists.
Fine tuning instead of training from scratch.
So once again
if you find a model that is almost what you need
but not quite fine tuning the last couple of layers
making it really fit your purpose instead
of training a large transformer
from scratch can really help,
starting with smaller experiments
and debugging as you go.
So that means, for example, training
figuring out data encoding, figuring out, you know
making sure that there's no small bugs, that you'll
you'll realize, you know, 16 hours into training
starting small and really making sure
that what you're doing, what your code is, is stable.
And then finally doing a literature review to
choose hyper parameter ranges and then following
up with a random search instead of a grid search.
So random searches for hyper parameters
combinations have actually shown to be as efficient
in finding the optimal configuration as grid search.
But obviously you're not trying all possible combinations
you're only trying a subset of them.
So this can really help as well.
So now if we go back
to the original paper by Strubell et all in 2019
the infamous five cars in their lifetimes paper.
If you just look
at a transformer of 200 million perimeter transformer
it is carbon footprint is around 200 pounds of CO2
which is significant
but it's nowhere near five cars, right?
It's not even a transatlantic flight.
How it really adds up is when you're doing
neural architecture search
when you're doing hyper parameter tuning, and
this is trying all possible combinations
et cetera, et cetera.
And this is where
like the 600,000 pounds of CO2 came from.
So this is really where things add up.
So, but if you're doing things mindfully and conscientiously
then your carbon footprint wont be as big as,
as the paper implied, some tools to figure
out how much CO2 exactly you're emitting.
There's a web-based tool called machine
learning submissions calculator, which allows you
to manually input, for example, which hardware you used
how many hours you used it for
where it was located locally or in the cloud.
And then it's gonna give you an estimate
of how much CO2 you emitted.
Another tool that does this programmatically,
is called code carbon.
So you can PIP install it, you can, you can go to the GitHub
and essentially it runs in parallel to your code.
So essentially you call it
and then you do all your training.
And then at the end it's gonna give you an estimate
a CSV file with an estimate of your emissions.
And it's gonna give you some comparisons.
It's got a visual UI where you can really look
at how this compares to driving a car or watching TV.
So it can give you an idea
of the scope of your emissions as well.
And actually, code carbon is already integrated into auto
and LP and hopefully people will be using it
out of the box and easily tracking their emissions all
through training and deploying transformers.