(air whooshing)
- How to slice and dice the dataset?
Most of the time, the data you work with
won't be perfectly prepared for training models.
In this video, we'll explore various features
that the datasets library provides to clean up your data.
The datasets library provides several built-in methods
that allow you to wrangle your data in various ways.
In this video, we'll see how you can shuffle
and split your data, select the rows you're interested in,
tweak the columns,
and apply processing functions with the map method.
Let's start with shuffling.
It is generally a good idea
to apply shuffling to your training set
so that your model doesn't learn
any artificial ordering the data.
If you wanna shuffle the whole dataset,
you can apply the appropriately named shuffle method.
You can see an example of this method in action here,
where we've downloaded the training split
of the squad dataset and shuffled all the rows randomly.
Another way to shuffle the data
is to create random train and test splits.
This can be useful if you have to create
your own test splits from raw data.
To do this, you just apply the train_test_split method
and specify how large the test split should be.
In this example, we specify that the test set should be 10%
of the total dataset size.
You can see that the output of the train_test_split method
is a DatasetDict object
whose keys correspond to the new splits.
Now that we know how to shuffle the dataset,
let's take a look at returning the rows we're interested in.
The most common way to do this is with the select method.
This method expects a list
or a generator of the datasets indices,
and will then return a new dataset object
containing just those rows.
If you wanna create a random sample of rows,
you can do this by chaining the shuffle
and select methods together.
In this example,
we've created a sample of five elements
from the squad dataset.
The last way to pick out specific rows in a dataset
is by applying the filter method.
This method checks whether each row
fulfills some condition or not.
For example, here we've created a small lambda function
that checks whether the title starts with the letter L.
Once we apply this function with the filter method,
we get a subset of the data just containing these rows.
So far, we've been talking about the rows of a dataset,
but what about the columns?
The datasets library has two main methods
for transforming columns,
a rename_column method to change the name of the column
and a remove_columns method to delete them.
You can see examples of both these methods here.
Some datasets have nested columns,
and you can expand these by applying the flatten method.
For example, in the squad dataset,
the answers column contains a text and answer_start field.
If we wanna promote them to their own separate columns,
we can apply flatten as shown here.
Now of course, no discussion of the datasets library
would be complete without mentioning the famous map method.
This method applies a custom processing function
to each row in the dataset.
For example, here we first define
a lowercase title function,
that simply lowercases the text in the title column.
And then we feed that function to the map method,
and voila, we now have lowercase titles.
The map method can also be used to feed batches of rows
to the processing function.
This is especially useful for tokenization
where the tokenizer is backed by the Tokenizers library,
and they can use fast multithreading
to process batches in parallel.
(air whooshing)