(air whooshing)
(logo popping)
(metal sliding)
- Memory mapping and streaming.
In this video, we'll take a look
at two core features of the Datasets library
that allow you to load and process huge datasets
without blowing up your laptop's CPU.
Nowadays, it's not uncommon to find yourself
working with multi-GB sized datasets,
especially if you're planning to pretrain
a transformer like BERT or GPT-2 from scratch.
In these cases, even loading the data can be a challenge.
For example, the c4 corpus used to pretrain T5
consists of over two terabytes of data.
To handle these large datasets,
the Datasets library is built on two core features:
the Apache Arrow format and a streaming API.
Arrow is designed for high-performance data processing
and represents each table-like dataset
with a column-based format.
As you can see in this example, column-based formats
group the elements of a table in consecutive blocks of RAM
and this unlocks fast access and processing.
Arrow is great at processing data at any scale
but some datasets are so large
that you can't even fit them on your hard disk.
So for these cases, the Datasets library provides
a streaming API that allows you to progressively download
the raw data one element at a time.
The result is a special object called an IterableDataset
that we'll see in more detail soon.
Let's start by looking at why Arrow is so powerful.
The first feature is that it treats every dataset
as a memory-mapped file.
Now, memory mapping is a mechanism
that maps a portion of a file or an entire file and disc
to a chunk of virtual memory.
This allows applications
to access segments of an extremely large file
without having to read the whole file into memory first.
Another cool feature of Arrow's memory mapping capabilities
is that it allows multiple processes
to work with the same large dataset
without moving it or copying it in any way.
This zero-copy feature of Arrow
makes it extremely fast for iterating over a dataset.
And this example, you can see that we iterate
over 15 million rows in about a minute
just using a standard laptop.
That's not too bad at all.
Let's now take a look at how we can stream a large dataset.
The only change you need to make is to set
the streaming=True argument in the load_dataset() function.
This will return a special IterableDataset object
which is a bit different to the Dataset objects
we've seen in other videos.
This object is an iterable,
which means we can't index it to access elements,
but instead we iterate on it
using the iter and next methods.
This will download and access
a single example from the dataset,
which means you can progressively iterate
through a huge dataset without having to download it first.
Tokenizing text with a map() method
also works in a similar way.
We first stream the dataset
and then apply the map() method with the tokenizer.
To get the first tokenized example, we apply iter and next.
The main difference with an IterableDataset is that
instead of using a select() method to return examples,
we use the take() and skip() methods
because we can't index into the dataset.
The take() method returns
the first N examples in the dataset,
while skip(), as you can imagine,
skips the first N and returns the rest.
You can see examples of both of these methods in action
where we create a validation set
from the first 1000 examples
and then skip those to create the training set.
(air whooshing)