(air whooshing)
- Supercharge your PyTorch training loop
with Hugging Face Accelerate.
There are multiple setups
on which you can run your training:
it could be on CPU, GPUs, TPUs,
distributed on one machine with several devices,
or even several machines, often called nodes,
each with multiple devices.
On top of that, there are new tweaks
to make your training faster or more efficient,
like mixed precision and DeepSpeed.
Each of those setups or training tweaks
requires you to change the code of your training loop
in one way or another and to learn a new API.
All those setups are handled by the Trainer API,
and there are several third-party libraries that can help.
The problem with those
is that they can feel like a black box
and that it might not be easy to implement the tweak
to the training loop you need.
Accelerate has been designed specifically
to let you retain full control over your training loop
and be as non-intrusive as possible.
With just four lines of code to add to your training loop,
here shown on the example of the training loop video,
Accelerate will handle all the setups
and training tweaks mentioned on the first slide.
It's only one API to learn and master
instead of 10 different ones.
More specifically, you have to import and instantiate
an accelerator object,
that will handle all the necessary code
for your specific setup.
Then you have to send it the model,
optimizer and dataloaders you are using
in the prepare method, which is the main method to remember.
Accelerate handles device placement,
so you don't need to put your batch
on the specific device you are using.
Finally, you have to replace the loss.backward line
by accelerator.backwardloss,
and that's all you need!
Accelerate also handles distributed evaluation.
You can still use a classic evaluation loop
such as the one we saw in the training loop video,
in which case all processes
will perform the full evaluation.
To use a distributed evaluation,
you just have to adapt your evaluation loop like this:
pass along the evaluation dataloader
to the accelerator.prepare method, like for training.
Then you can dismiss the line
that places the batch on the proper device,
and just before passing your predictions
and labels to your metric, use accelerator.gather
to gather together the predictions
and labels from each process.
A distributed training script
has to be launched several times on different processes,
for instance, one per GPU you are using.
You can use the PyTorch tools to do that
if you are familiar with them,
but Accelerate also provides an easy API
to configure your setup and launch your training script.
In a terminal, run accelerate config
and answer the small questionnaire
to generate a configuration file
with all the relevant information,
then you can just run accelerate launch,
followed by the path to your training script.
In a notebook, you can use the notebook launcher function
to launch your training.
(air whooshing)