(subtle blast)
- The post-processing step in a question-answering task.
When doing question answering,
the processing of the initial dataset
implies splitting examples in several features,
which may or may not contain the answer.
Passing those features through the model
will give us logits for the start and end positions,
since our labels are the indices of the tokens
that correspond to the start and end the answer.
We must then somehow convert those logits into an answer,
and then pick one of the various answers each feature gives
to be the answer for a given example.
For the processing step,
you should refer to the video linked below.
It's not very different for validation,
we just need to add a few lines to keep track of two things:
instead of discarding the offset mappings,
we keep them, and also include in them the information
of where the context is by setting the offsets
of the special tokens and the question to None.
Then we also keep track of the example ID for each feature,
to be able to map back feature
to the examples that they originated from.
If you don't want to compute the validation loss,
you won't need to include all the special code
that we used to create the labels.
With this done, we can apply that preprocessing function
using the map method.
We take the SQUAD dataset like in the preprocessing
for question-answering video.
Once this is done, the next step is to create our model.
We use the default model behind
the question-answering pipeline here,
but you should use any model you want to evaluate.
With the to_tf_dataset method,
we can just sent our processed dataset to model.predict,
and we directly get our start and end logits
for the whole dataset as NumPy arrays.
With this done, we can really dive into the post-processing.
First, we'll need a map from example to features,
which we can create like this.
Now, for the main part of the post-processing,
let's see how to extract an answer from the logits.
We could just take the best index
for the start and end logits and be done,
but if our model predicts something impossible,
like tokens in the questions,
we will look at more of the logits.
Note that in the question-answering pipeline,
we attributed the score to each answer
based on the probabilities, which we did not compute here.
In terms of logits, the multiplication we had
in the scores becomes an addition.
To go fast, we don't look
at all possible start and end logits,
but the 20 best ones.
We ignore the logits that spawn impossible answers
or answer that are too long.
As we saw in the preprocessing,
the label "0, 0" correspond to no answer,
otherwise we use the offset
to get the answer inside the context.
Let's have a look at the predicted answer
for the first feature,
which is the answer with the best score,
or the best logit score since the SoftMax
is an increasing function.
The model got it right.
Next, we just have to loop this for every example,
picking for each the answer with the best logit score
in all the features the example generated.
Now you know how to get answers from your model predictions.
(subtle blast)