(transition music)
- Saving and reloading a dataset.
In this video, we'll take a look
at saving a dataset in various formats
and explore the ways to reload the saved data.
When you download a dataset, the processing scripts and data
are stored locally on your computer.
The cache allows the Datasets library
to avoid re-downloading
or processing the entire dataset every time you use it.
Now, the data is stored in the form of Arrow tables
whose location can be found
by accessing the dataset's cache_files attribute.
In this example, we've downloaded the allocine dataset
from the Hugging Face Hub, and you can see
that there are three Arrow files
stored in the cache, one for each split.
But in many cases, you'll wanna save your dataset
in a different location or format.
As shown in the table, the Datasets library
provides four main functions to achieve this.
Now, you're probably already familiar
with the CSV and JSON formats, both of which are great
if you just wanna quickly save a small
or medium-sized dataset.
But if your dataset is huge,
you'll wanna save it in either the Arrow or Parquet formats.
Arrow files are great if you plan to reload
or process the data in the near future.
While Parquet files are designed
for long-term storage and are very space-efficient.
Let's take a closer look at each format.
To save a dataset or a dataset_dict object
in the Arrow format, we use the save_to_disk function.
As you can see in this example,
we simply provide the path we wish to save the data to
and the Datasets library
will automatically create a directory
for each split to store the Arrow table and the metadata.
Since we're dealing with a dataset_dict object
that has multiple splits,
this information is also stored
in the dataset_dict.json file.
Now, when we wanna reload the Arrow datasets,
we use the load_from_disk function.
We simply pass the path of our dataset directory,
and voila, the original dataset is recovered.
If we wanna save our dataset
in the CSV format, we use the to_csv function.
In this case, you'll need to loop
over the splits of the dataset_dict object
and save each dataset as an individual CSV file.
Since the to_csv function is based on the one from Pandas,
you can pass keyword arguments to configure the output.
In this example, we've set the index argument
to None to prevent the dataset's index column
from being included in the CSV files.
To reload our CSV files,
we just then use the familiar load_dataset function
together with the CSV loading script
and the data_files argument,
which specifies the file names associated with each split.
As you can see in this example,
by providing all the splits and their file names,
we've recovered the original dataset_dict object.
Now, to save a dataset in the JSON
or Parquet formats is very similar to the CSV case.
We use either the to_json function for JSON files
or the to_parquet function for Parquet ones.
And just like the CSV case, we need to loop over the splits
to save each one as an individual file.
And once our datasets are saved as JSON or Parquet files,
we can reload them again
with the appropriate script in the load_dataset function.
And we just need to provide a data_files argument as before.
This example shows
how we can reload our save datasets in either format.
And with that, you now know
how to save your datasets in various formats.
(transition music)