- Let's study how to preprocess a dataset
for question answering.
Question answering is a task
of finding answers to a question in some context.
For example, we'll use the SQuAD dataset
in which we remove columns we won't use
and just extract the information we will need
for the labels,
the start and the end of the answer in the context.
If you have your own dataset for question answering,
just make sure you clean your data to get to the same point,
with one column containing the questions,
one column containing the context,
one column for the index of the start and end character
of the answer in the context.
Note that the answer must be part of the context.
If you want to perform generative question answering,
look at one of the sequence to sequence videos linked below.
Now, if we have a look at the tokens
we will feed our model,
we'll see the answer lies somewhere inside the context.
For very long context, that answer may get truncated
by the tokenizer.
In this case, we won't have any proper labels for our model,
so we should keep the truncated part
as a separate feature instead of discarding it.
The only thing we need to be careful with
is to allow some overlap between separate chunks
so that the answer is not truncated
and that the feature containing the answer
gets sufficient context to be able to predict it.
Here is how it can be done by the tokenizer.
We pass it the question, context, set a truncation
for the context only, and the padding to the maximum length.
The stride argument is where we set the number
of overlapping tokens,
and the return overflowing tokens equals true
means we don't want to discard the truncated part.
Lastly, we also return the offset mappings
to be able to find the tokens corresponding
to the answer start and end.
We want those tokens because they will be the labels
we pass through our model.
In a one-hot encoded version, here is what they look like.
If the context we have does not contain the answer,
we set the two labels to the index of the CLS token.
We also do this if the context
only partially contains the answer.
In terms of code, here is how we can do it.
Using the sequence IDs of an input,
we can determine the beginning and the end of the context.
Then, we know if we have to return to the CLS position
for the two labels or we determine the position
of the first and last tokens of the answer.
We can check it works properly on our previous example.
Putting it all together looks like this big function,
which we can apply to our datasets with the map method.
Since we applied padding during the tokenization,
we can then use this directly as the trainer
or apply the to_tf_dataset method to use Keras.fit.