(screen whooshing)
(sticker popping)
(screen whooshing)
- Loading a custom dataset.
Although the Hugging Face Hub hosts
over a thousand public datasets,
you'll often need to work with data
that is stored on your laptop or some remote server.
In this video, we'll explore how the Datasets library
can be used to load datasets that aren't available
on the Hugging Face Hub.
As you can see in this table, the Datasets library
provides several in-built scripts
to load datasets in several formats.
To load a dataset in one of these formats,
you just need to provide the name of the format
to the load_dataset function,
along with a data_files argument
that points to one or more filepaths or URLs.
To see this in action, let's start by loading a CSV file.
In this example, we first download a dataset
about wine quality from the UCI machine learning repository.
Since this is a CSV file, we then specify
the CSV loading script.
Now, this script needs to know where our data is located,
so we provide the filename
as part of the data_files argument.
And the loading script also allows you
to pass several keyword arguments,
so here we've also specified
that the separator is a semi-colon.
And with that, we can see the dataset
is loaded automatically as a DatasetDict object,
with each column in the CSV file represented as a feature.
If your dataset is located on some remote server like GitHub
or some other repository,
the process is actually very similar.
The only difference is that now the data_files argument
points to a URL instead of a local filepath.
Let's now take a look at loading raw text files.
This format is quite common in NLP,
and you'll typically find books and plays
are just a single file with raw text inside.
In this example, we have a text file of Shakespeare plays
that's stored on a GitHub repository.
And as we did for CSV files,
we simply choose the text loading script
and point the data_files argument to the URL.
As you can see, these files are processed line-by-line,
so empty lines in the raw text are also represented
as a row in the dataset.
For JSON files, there are two main formats to know about.
The first one is called JSON Lines,
where every row in the file is a separate JSON object.
For these files, you can load the dataset
by selecting the JSON loading script
and pointing the data_files argument to the file or URL.
In this example, we've loaded a JSON lines files
based on Stack Exchange questions and answers.
The other format is nested JSON files.
These files basically look like one huge dictionary,
so the load_dataset function allow you to specify
which specific key to load.
For example, the SQuAD dataset for question and answering
has its format, and we can load it by specifying
that we're interested in the data field.
There is just one last thing to mention
about all of these loading scripts.
You can have more than one split,
you can load them by treating data files as a dictionary,
and map each split name to its corresponding file.
Everything else stays completely unchanged
and you can see an example of loading
both the training and validation splits for this SQuAD here.
And with that, you can now load datasets from your laptop,
the Hugging Face Hub, or anywhere else want.
(screen whooshing)