- Let's have a look inside the question answering pipeline.
The question answering pipeline can extracts answers
to questions from a given context or passage of text,
like this part of the transformers repo README.
It also works for very long contexts,
even if the answer is at the very end, like in this example.
In this video, we will see why.
The question answering pipeline follows the same steps
as the other pipelines:
the question and context are tokenized as a sentence pair,
fed to the model then some post-processing is applied.
The tokenization and model steps should be familiar.
We use the auto class suitable for question answering
instead of sequence classification,
but one key difference with text classification
is that our model outputs two tensors named start logits
and end logits.
Why is that?
Well, this is the way the model finds the answer
to the question.
First, let's have a look at the model inputs.
Its numbers associated with the tokenization
of the question followed by the context
with the usual CLS and SEP special tokens.
The answer is a part of those tokens.
So we ask the model to predict which token starts
the answer and which ends the answer.
For our two logit outputs,
the theoretical labels are the pink and purple vectors.
To convert those logits into probabilities,
we will need to apply a SoftMax,
like in the text classification pipeline.
We just mask the tokens that are not part of the context
before doing that, leaving the initial CLS token unmasked
as we use it to predict an impossible answer.
This is what it looks in terms of code.
We use a large negative number for the masking,
since its exponential will then be zero.
Now, the probability for each start
and end position corresponding to a possible answer,
we give a score that is the product
of the start probabilities and end probabilities
at those positions.
Of course, a start index greater than an end index
corresponds to an impossible answer.
Here is the code to find the best score
for a possible answer.
Once we have the start and end positions of the tokens,
we use the offset mappings provided by our tokenizer
to find the span of characters in the initial context,
and get our answer.
Now, when the context is long, it might get truncated
by the tokenizer.
This might result in part of the answer, or worse,
the whole answer, being truncated.
So we don't discard the truncated tokens
but build new features with them.
Each of those features contains the question,
then a chunk of text in the context.
If we take disjoint chunks of texts,
we might end up with the answer being split
between two features.
So instead, we take overlapping chunks of texts,
to make sure at least one of the chunks will fully contain
the answer to the question.
The tokenizers do all of this for us automatically
with the return overflowing tokens option.
The stride argument controls
the number of overlapping tokens.
Here is how our very long context gets truncated
in two features with some overlap.
By applying the same post-processing we saw before
for each feature, we get the answer with a score
for each of them,
and we take the answer with the best score
as a final solution.