(slide whooshes)
- The Hugging Face Datasets library, a quick overview.
The Hugging Face Datasets library
is a library that provides an API to quickly download
many public datasets and preprocess them.
In this video we will explore how to do that.
The downloading part is easy,
with the load_dataset function.
You can directly download and cache a dataset
from its identifier on the Dataset hub.
Here, we fetch the MRPC dataset from the GLUE benchmark,
which is a dataset containing pairs of sentences
where the task is to determine the paraphrases.
The object returned by the load_dataset function
is a DatasetDict, which is a sort of dictionary
containing each split of our dataset.
We can access each split by indexing with its name.
This split is then an instance of the Dataset class,
with columns, here sentence1, sentence2,
label and idx, and rows.
We can access a given element by its index.
The amazing thing about the Hugging Face Datasets library
is that everything is saved to disk using Apache Arrow,
which means that even if your dataset is huge,
you won't get out of RAM.
Only the elements you request are loaded in memory.
Accessing a slice of your dataset is as easy as one element.
The result is then a dictionary with list of values
for each keys.
Here the list of labels,
the list of first sentences
and the list of second sentences.
The features attribute of a Dataset
gives us more information about its columns.
In particular, we can see here
it gives us the correspondence
between the integers and names for the labels.
Zero stands for not equivalent and one for equivalent.
To preprocess all the elements of our dataset,
we need to tokenize them.
Have a look at the video "Preprocess sentence pairs"
for a refresher,
but you just have to send the two sentences
to the tokenizer with some additional keyword arguments.
Here we indicate a maximum length of 128
and pad inputs shorter than this length,
truncate inputs that are longer.
We put all of this in a tokenize_function
that we can directly apply to all the splits in our dataset
with the map method.
As long as the function returns a dictionary-like object,
the map method will add new columns as needed
or update existing ones.
To speed up preprocessing
and take advantage of the fact our tokenizer
is backed by Rust,
thanks to the Hugging Face Tokenizers library,
we can process several elements at the same time
to our tokenize function, using the batched=True argument.
Since the tokenizer can handle
list of first sentences, list of second sentences,
the tokenize_function does not need to change for this.
You can also use multiprocessing with the map method.
Check out its documentation in the linked video.
Once this is done, we are almost ready for training.
We just remove the columns we don't need anymore
with the remove_columns method,
rename label to labels,
since the models from the Hugging Face Transformers
library expect that,
and set the output format to our desired backend,
Torch, TensorFlow or NumPy.
If needed, we can also generate a short sample of a dataset
using the select method.
(slide whooshes)