notebooks/question_answering.ipynb (1,691 lines of code) (raw):
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "rEJBSTyZIrIb"
},
"source": [
"# Question-Answering on IPU using RoBERTa - Fine-tuning"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"In this notebook, we will see how to fine-tune one of the [🤗 Transformers](https://github.com/huggingface/transformers) models to a question-answering task. Question answering involves extracting the answer to a question from a given context. We will see how to easily load a dataset for these kinds of tasks and use the `IPUTrainer` API to fine-tune a model on it.\n",
"\n",
"\n",
"\n",
"**Note:** This notebook fine-tunes models that answer questions by taking a substring of a context, not by generating new text."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"| Domain | Tasks | Model | Datasets | Workflow | Number of IPUs | Execution time |\n",
"|---------|-------|-------|----------|----------|--------------|--------------|\n",
"| Natural language processing | Classification | roberta-base | SQUADv1 | Fine-tuning | 4 | 21min |\n",
"\n",
"[](https://www.graphcore.ai/join-community)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Environment setup\n",
"\n",
"The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n",
"\n",
"[](https://ipu.dev/3IutIto)\n",
"\n",
"To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dependencies and configuration\n",
"\n",
"In order to improve usability and support for future users, Graphcore would like to collect information about the\n",
"applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n",
"\n",
"- User progression through the notebook\n",
"- Notebook details: number of cells, code being run and the output of the cells\n",
"- Environment details\n",
"\n",
"You can disable logging at any time by running `%unload_ext graphcore_cloud_tools.notebook_logging.gc_logger` from any cell."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Install the dependencies for this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install \"optimum-graphcore==0.7\"\n",
"%pip install graphcore-cloud-tools[logger]@git+https://github.com/graphcore/graphcore-cloud-tools\n",
"%load_ext graphcore_cloud_tools.notebook_logging.gc_logger"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook is built to run on any question-answering task with the same format as SQUAD (version 1 or 2), with any model checkpoint from the [🤗 Model Hub](https://huggingface.co/models) as long as that model has a version with a token classification head and a fast tokenizer (refer to the [supported frameworks table](https://huggingface.co/docs/transformers/index#supported-frameworks) to check). The model must also be supported by Optimum Graphcore. The IPU config files of the supported models are available in Graphcore's [🤗 account](https://huggingface.co/Graphcore). You can also create your own IPU config file locally. You might need to make some small adjustments if you choose to use a different dataset than the one used here.\n",
"\n",
"In this notebook, we are using both data parallelism and pipeline parallelism (see this [tutorial on efficient data loading](https://github.com/graphcore/examples/tree/master/tutorials/pytorch/efficient_data_loading)). Therefore the global batch size, which is the actual number of samples used for the weight update, is determined with three factors:\n",
"- `global_batch_size = micro_batch_size * gradient_accumulation_steps * replication_factor`,\n",
"\n",
"The replication factor is determined by the IPU Pod type, which will be used as a key to select the replication factor from a dictionary defined in the IPU config file. For example, the dictionary in the IPU config file [Graphcore/roberta-base-ipu](https://huggingface.co/Graphcore/roberta-base-ipu/blob/main/ipu_config.json) looks like this:,\n",
"`\"replication_factor\": {\"pod4\": 1, \"pod8\": 2, \"pod16\": 4, \"pod32\": 8, \"pod64\": 16, \"default\": 1}`\n",
" \n",
"Depending on your model and the IPU Pod you are using, you might need to adjust these three batch-size-related arguments."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# This flag is the difference between SQUAD v1 or 2 (if you're using another dataset, it indicates if impossible\n",
"# answers are allowed or not).\n",
"squad_v2 = False\n",
"model_checkpoint = \"roberta-base\"\n",
"\n",
"ipu_config_name = \"Graphcore/roberta-base-ipu\"\n",
"micro_batch_size = 1\n",
"gradient_accumulation_steps = 32"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Values for machine size and cache directories can be configured through environment variables or directly in the notebook:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"n_ipu = int(os.getenv(\"NUM_AVAILABLE_IPU\", 4))\n",
"executable_cache_dir = os.getenv(\"POPLAR_EXECUTABLE_CACHE_DIR\", \"/tmp/exe_cache/\") + \"/question_answering\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Sharing your model with the Community"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can share your model with the 🤗 community. You do this by completing the following steps:\n",
"\n",
"1. Store your authentication token from the 🤗 website. [Sign up to 🤗](https://huggingface.co/join) if you haven't already.\n",
"2. Execute the following cell and input your username and password."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from huggingface_hub import notebook_login\n",
"\n",
"notebook_login()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Then you need to install Git-LFS to manage large files:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!apt install git-lfs"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "whPRbBNbIrIl"
},
"source": [
"## Loading the dataset"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "W7QYTpxXIrIl"
},
"source": [
"We will use the [🤗 Datasets](https://github.com/huggingface/datasets) library to download the data and get the metric we will use for evaluation (to compare our model to the benchmark). This can be easily done with the functions `load_dataset` and `load_metric`. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "IreSlFmlIrIm"
},
"outputs": [],
"source": [
"from datasets import load_dataset, load_metric"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "CKx2zKs5IrIq"
},
"source": [
"For our example here, we'll use the [SQUAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). The notebook should work with any question-answering dataset provided by the 🤗 Datasets library. If you're using your own dataset defined from a JSON or CSV file (see the Datasets documentation for [loading data from local files](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files)), the notebook might need some adjustments in the names of the columns used."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 270,
"referenced_widgets": [
"69caab03d6264fef9fc5649bffff5e20",
"3f74532faa86412293d90d3952f38c4a",
"50615aa59c7247c4804ca5cbc7945bd7",
"fe962391292a413ca55dc932c4279fa7",
"299f4b4c07654e53a25f8192bd1d7bbd",
"ad04ed1038154081bbb0c1444784dcc2",
"7c667ad22b5740d5a6319f1b1e3a8097",
"46c2b043c0f84806978784a45a4e203b",
"80e2943be35f46eeb24c8ab13faa6578",
"de5956b5008d4fdba807bae57509c393",
"931db1f7a42f4b46b7ff8c2e1262b994",
"6c1db72efff5476e842c1386fadbbdba",
"ccd2f37647c547abb4c719b75a26f2de",
"d30a66df5c0145e79693e09789d96b81",
"5fa26fc336274073abbd1d550542ee33",
"2b34de08115d49d285def9269a53f484",
"d426be871b424affb455aeb7db5e822e",
"160bf88485f44f5cb6eaeecba5e0901f",
"745c0d47d672477b9bb0dae77b926364",
"d22ab78269cd4ccfbcf70c707057c31b",
"d298eb19eeff453cba51c2804629d3f4",
"a7204ade36314c86907c562e0a2158b8",
"e35d42b2d352498ca3fc8530393786b2",
"75103f83538d44abada79b51a1cec09e",
"f6253931d90543e9b5fd0bb2d615f73a",
"051aa783ff9e47e28d1f9584043815f5",
"0984b2a14115454bbb009df71c1cf36f",
"8ab9dfce29854049912178941ef1b289",
"c9de740e007141958545e269372780a4",
"cbea68b25d6d4ba09b2ce0f27b1726d5",
"5781fc45cf8d486cb06ed68853b2c644",
"d2a92143a08a4951b55bab9bc0a6d0d3",
"a14c3e40e5254d61ba146f6ec88eae25",
"c4ffe6f624ce4e978a0d9b864544941a",
"1aca01c1d8c940dfadd3e7144bb35718",
"9fbbaae50e6743f2aa19342152398186",
"fea27ca6c9504fc896181bc1ff5730e5",
"940d00556cb849b3a689d56e274041c2",
"5cdf9ed939fb42d4bf77301c80b8afca",
"94b39ccfef0b4b08bf2fb61bb0a657c1",
"9a55087c85b74ea08b3e952ac1d73cbe",
"2361ab124daf47cc885ff61f2899b2af",
"1a65887eb37747ddb75dc4a40f7285f2",
"3c946e2260704e6c98593136bd32d921",
"50d325cdb9844f62a9ecc98e768cb5af",
"aa781f0cfe454e9da5b53b93e9baabd8",
"6bb68d3887ef43809eb23feb467f9723",
"7e29a8b952cf4f4ea42833c8bf55342f",
"dd5997d01d8947e4b1c211433969b89b",
"2ace4dc78e2f4f1492a181bcd63304e7",
"bbee008c2791443d8610371d1f16b62b",
"31b1c8a2e3334b72b45b083688c1a20c",
"7fb7c36adc624f7dbbcb4a831c1e4f63",
"0b7c8f1939074794b3d9221244b1344d",
"a71908883b064e1fbdddb547a8c41743",
"2f5223f26c8541fc87e91d2205c39995"
]
},
"id": "s_AY1ATSIrIq",
"outputId": "fd0578d1-8895-443d-b56f-5908de9f1b6b"
},
"outputs": [],
"source": [
"datasets = load_dataset(\"squad_v2\" if squad_v2 else \"squad\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "RzfPtOMoIrIu"
},
"source": [
"The `datasets` object is [`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict), which contains one key for the training, validation and test sets."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GWiVUF0jIrIv",
"outputId": "35e3ea43-f397-4a54-c90c-f2cf8d36873e"
},
"outputs": [],
"source": [
"datasets"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see the training, validation and test sets all have a column for the context, the question and the answers to those questions."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "u3EtYfeHIrIz"
},
"source": [
"To access an actual element, you need to select a split (\"train\" in the example) and then specify an index:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "X6HrpprwIrIz",
"outputId": "d7670bc0-42e4-4c09-8a6a-5c018ded7d95"
},
"outputs": [],
"source": [
"datasets[\"train\"][0]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see the answers are indicated by their start position in the text (here at character 515) and their full text, which is a substring of the context as we mentioned above."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "WHUmphG3IrI3"
},
"source": [
"We want to get a sense of what the data looks like, so we define the `show_random_elements` function to display some samples picked randomly from the dataset (automatically decoding the labels in passing)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "i3j8APAoIrI3"
},
"outputs": [],
"source": [
"from datasets import ClassLabel, Sequence\n",
"import random\n",
"import pandas as pd\n",
"from IPython.display import display, HTML\n",
"\n",
"def show_random_elements(dataset, num_examples=10):\n",
" assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
" picks = []\n",
" for _ in range(num_examples):\n",
" pick = random.randint(0, len(dataset)-1)\n",
" while pick in picks:\n",
" pick = random.randint(0, len(dataset)-1)\n",
" picks.append(pick)\n",
" \n",
" df = pd.DataFrame(dataset[picks])\n",
" for column, typ in dataset.features.items():\n",
" if isinstance(typ, ClassLabel):\n",
" df[column] = df[column].transform(lambda i: typ.names[i])\n",
" elif isinstance(typ, Sequence) and isinstance(typ.feature, ClassLabel):\n",
" df[column] = df[column].transform(lambda x: [typ.feature.names[i] for i in x])\n",
" display(HTML(df.to_html()))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SZy5tRB_IrI7",
"outputId": "ba8f2124-e485-488f-8c0c-254f34f24f13",
"scrolled": true
},
"outputs": [],
"source": [
"show_random_elements(datasets[\"train\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "n9qywopnIrJH"
},
"source": [
"## Preprocessing the training data"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "YVx71GdAIrJH"
},
"source": [
"Before we can feed this data to our model, we need to do some preprocessing. This is done by a 🤗 Transformers `Tokenizer` which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pre-trained vocabulary) and put it in a format the model expects, as well as generate the other inputs that the model requires.\n",
"\n",
"To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n",
"\n",
"- We get a tokenizer that corresponds to the model architecture we want to use.\n",
"- We download the vocabulary used when pre-training this specific checkpoint.\n",
"\n",
"That vocabulary will be cached, so it's not downloaded again the next time we run the cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eXNLu_-nIrJI"
},
"outputs": [],
"source": [
"from transformers import AutoTokenizer\n",
" \n",
"tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "Vl6IidfdIrJK"
},
"source": [
"The following assertion ensures that our tokenizer is a fast tokenizer (backed by Rust) from the 🤗 Tokenizers library. Fast tokenizers are available for almost all models, and we will need some of the special features they have for our preprocessing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import transformers\n",
"assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can check which type of models have a fast tokenizer available and which don't in the [supported frameworks table](https://huggingface.co/docs/transformers/index#supported-frameworks)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "rowT4iCLIrJK"
},
"source": [
"You can directly call this tokenizer on two sentences (one for the question, one for the context):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a5hBlsrHIrJL",
"outputId": "acdaa98a-a8cd-4a20-89b8-cc26437bbe90"
},
"outputs": [],
"source": [
"tokenizer(\"What is your name?\", \"My name is Sylvain.\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here, but they are required by the model we will instantiate later. You can learn more about this in this [tutorial on preprocessing](https://huggingface.co/transformers/preprocessing.html).\n",
"\n",
"One topic that needs careful attention when preprocessing the data in question answering is how to deal with very long documents. In other tasks, we usually truncate them when they are longer than the model maximum sentence length. With question answering this is not feasible because removing part of the the context might result in losing the answer we are looking for. To deal with this, we will allow one (long) sample in our dataset to give several input features, each of length shorter than the maximum length of the model (or the one we set as a hyper-parameter). Also, just in case the answer lies at the point we split a long context, we allow some overlap between the features we generate controlled by the hyper-parameter `doc_stride`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"max_length = 384 # The maximum length of a feature (question and context)\n",
"doc_stride = 128 # The authorized overlap between two part of the context when splitting it is needed."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's find one long sample in our dataset:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for i, example in enumerate(datasets[\"train\"]):\n",
" if len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"]) > 384:\n",
" break\n",
"example = datasets[\"train\"][i]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Without any truncation, we get the following length for the input IDs:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(tokenizer(example[\"question\"], example[\"context\"])[\"input_ids\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, if we just truncate, we will lose information (and possibly the answer to our question):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"len(tokenizer(example[\"question\"], example[\"context\"], max_length=max_length, truncation=\"only_second\")[\"input_ids\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that we never want to truncate the question, only the context and we use the option `only_second` for the [truncation](https://huggingface.co/docs/transformers/pad_truncation#padding-and-truncation) argument. Now, our tokenizer can automatically return a list of features capped by a certain maximum length, with the overlap we talked above. We do this with `return_overflowing_tokens=True` and by passing the stride:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tokenized_example = tokenizer(\n",
" example[\"question\"],\n",
" example[\"context\"],\n",
" max_length=max_length,\n",
" truncation=\"only_second\",\n",
" return_overflowing_tokens=True,\n",
" stride=doc_stride\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we don't have one list of `input_ids`, but several: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"[len(x) for x in tokenized_example[\"input_ids\"]]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"If we decode them, we can see the overlap:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for x in tokenized_example[\"input_ids\"][:2]:\n",
" print(tokenizer.decode(x))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This means we have to do some work to properly treat the answers: we need to find in which of those features the answer actually is, and where exactly in that feature. The models we will use require the start and end positions of these answers in the tokens, so we will also need to map parts of the original context to some tokens. Thankfully, the tokenizer we're using can help us with that by returning an `offset_mapping`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tokenized_example = tokenizer(\n",
" example[\"question\"],\n",
" example[\"context\"],\n",
" max_length=max_length,\n",
" truncation=\"only_second\",\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" stride=doc_stride\n",
")\n",
"print(tokenized_example[\"offset_mapping\"][0][:100])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This gives, for each index of our input ID list, the corresponding start and end character in the original text that gave our token. The very first token (`[CLS]`) has (0, 0) because it doesn't correspond to any part of the question/answer, then the second token is the same as the characters 0 to 3 of the question:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"first_token_id = tokenized_example[\"input_ids\"][0][1]\n",
"offsets = tokenized_example[\"offset_mapping\"][0][1]\n",
"print(tokenizer.convert_ids_to_tokens([first_token_id])[0], example[\"question\"][offsets[0]:offsets[1]])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"So we can use this mapping to find the position of the start and end tokens of our answer in a given feature. We just have to distinguish which parts of the offsets correspond to the question and which parts correspond to the context, this is where the `sequence_ids` method of `tokenized_example` can be useful:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"sequence_ids = tokenized_example.sequence_ids()\n",
"print(sequence_ids)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"It returns `None` for the special tokens, then 0 or 1 depending on whether the corresponding token comes from the first sentence (the question) or the second (the context). Now with all of this, we can find the first and last token of the answer in one of our input features (or if the answer is not in this feature):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"answers = example[\"answers\"]\n",
"start_char = answers[\"answer_start\"][0]\n",
"end_char = start_char + len(answers[\"text\"][0])\n",
"\n",
"# Start token index of the current span in the text.\n",
"token_start_index = 0\n",
"while sequence_ids[token_start_index] != 1:\n",
" token_start_index += 1\n",
"\n",
"# End token index of the current span in the text.\n",
"token_end_index = len(tokenized_example[\"input_ids\"][0]) - 1\n",
"while sequence_ids[token_end_index] != 1:\n",
" token_end_index -= 1\n",
"\n",
"# Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).\n",
"offsets = tokenized_example[\"offset_mapping\"][0]\n",
"if (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
" # Move the token_start_index and token_end_index to the two ends of the answer.\n",
" # Note: we could go after the last offset if the answer is the last word (edge case).\n",
" while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
" token_start_index += 1\n",
" start_position = token_start_index - 1\n",
" while offsets[token_end_index][1] >= end_char:\n",
" token_end_index -= 1\n",
" end_position = token_end_index + 1\n",
" print(start_position, end_position)\n",
"else:\n",
" print(\"The answer is not in this feature.\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"And we can double check that it is indeed the theoretical answer:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(tokenizer.decode(tokenized_example[\"input_ids\"][0][start_position: end_position+1]))\n",
"print(answers[\"text\"][0])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"For this notebook to work with any kind of models, we need to account for the special case where the model expects padding on the left (in which case we switch the order of the question and the context):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pad_on_right = tokenizer.padding_side == \"right\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's put everything together in one function that we will apply to our training set. In the case of impossible answers (the answer is in another feature given by a sample with a long context), we set the CLS index for both the start and end position. We could also simply discard those samples from the training set if the flag `allow_impossible_answers` is False. Since the preprocessing is already complex enough as it is, we've kept it simple for this part."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def prepare_train_features(examples):\n",
" # Some of the questions have lots of whitespace on the left, which is not useful and will make the\n",
" # truncation of the context fail (the tokenized question will take a lots of space). So we remove that\n",
" # left whitespace\n",
" examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
"\n",
" # Tokenize our examples with truncation and padding, but keep the overflows using a stride. This results\n",
" # in one example possible giving several features when a context is long, each of those features having a\n",
" # context that overlaps a bit the context of the previous feature.\n",
" tokenized_examples = tokenizer(\n",
" examples[\"question\" if pad_on_right else \"context\"],\n",
" examples[\"context\" if pad_on_right else \"question\"],\n",
" truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
" max_length=max_length,\n",
" stride=doc_stride,\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" padding=\"max_length\",\n",
" )\n",
"\n",
" # Since one example might give us several features if it has a long context, we need a map from a feature to\n",
" # its corresponding example. This key gives us just that.\n",
" sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
" # The offset mappings will give us a map from token to character position in the original context. This will\n",
" # help us compute the start_positions and end_positions.\n",
" offset_mapping = tokenized_examples.pop(\"offset_mapping\")\n",
"\n",
" # Let's label those examples!\n",
" tokenized_examples[\"start_positions\"] = []\n",
" tokenized_examples[\"end_positions\"] = []\n",
"\n",
" for i, offsets in enumerate(offset_mapping):\n",
" # We will label impossible answers with the index of the CLS token.\n",
" input_ids = tokenized_examples[\"input_ids\"][i]\n",
" cls_index = input_ids.index(tokenizer.cls_token_id)\n",
"\n",
" # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n",
" sequence_ids = tokenized_examples.sequence_ids(i)\n",
"\n",
" # One example can give several spans, this is the index of the example containing this span of text.\n",
" sample_index = sample_mapping[i]\n",
" answers = examples[\"answers\"][sample_index]\n",
" # If no answers are given, set the cls_index as answer.\n",
" if len(answers[\"answer_start\"]) == 0:\n",
" tokenized_examples[\"start_positions\"].append(cls_index)\n",
" tokenized_examples[\"end_positions\"].append(cls_index)\n",
" else:\n",
" # Start/end character index of the answer in the text.\n",
" start_char = answers[\"answer_start\"][0]\n",
" end_char = start_char + len(answers[\"text\"][0])\n",
"\n",
" # Start token index of the current span in the text.\n",
" token_start_index = 0\n",
" while sequence_ids[token_start_index] != (1 if pad_on_right else 0):\n",
" token_start_index += 1\n",
"\n",
" # End token index of the current span in the text.\n",
" token_end_index = len(input_ids) - 1\n",
" while sequence_ids[token_end_index] != (1 if pad_on_right else 0):\n",
" token_end_index -= 1\n",
"\n",
" # Detect if the answer is out of the span (in which case this feature is labelled with the CLS index).\n",
" if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):\n",
" tokenized_examples[\"start_positions\"].append(cls_index)\n",
" tokenized_examples[\"end_positions\"].append(cls_index)\n",
" else:\n",
" # Otherwise move the token_start_index and token_end_index to the two ends of the answer.\n",
" # Note: we could go after the last offset if the answer is the last word (edge case).\n",
" while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:\n",
" token_start_index += 1\n",
" tokenized_examples[\"start_positions\"].append(token_start_index - 1)\n",
" while offsets[token_end_index][1] >= end_char:\n",
" token_end_index -= 1\n",
" tokenized_examples[\"end_positions\"].append(token_end_index + 1)\n",
"\n",
" return tokenized_examples"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "0lm8ozrJIrJR"
},
"source": [
"This function works with one or several samples. In the case of several samples, the tokenizer will return a list of lists for each key:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-b70jh26IrJS",
"outputId": "acd3a42d-985b-44ee-9daa-af5d944ce1d9"
},
"outputs": [],
"source": [
"features = prepare_train_features(datasets['train'][:5])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "zS-6iXTkIrJT"
},
"source": [
"To apply this function on all the sentences (or pairs of sentences) in our dataset, we just use the `map` method of the `dataset` object we created earlier. This will apply the function to all the elements of all the splits in `dataset`, so our training, validation and testing data will be preprocessed with a single command. Since our preprocessing changes the number of samples, we need to remove the old columns when applying it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DDtsaJeVIrJT",
"outputId": "aa4734bf-4ef5-4437-9948-2c16363da719"
},
"outputs": [],
"source": [
"tokenized_datasets = datasets.map(prepare_train_features, batched=True, remove_columns=datasets[\"train\"].column_names)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "voWiw8C7IrJV"
},
"source": [
"Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library detects when the function you pass to `map` has changed (and thus requires not using the cached data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files. You can pass `load_from_cache_file=False` in the call to `map` to not use the cached files and force the preprocessing to be applied again.\n",
"\n",
"Note that we passed `batched=True` to encode the text sample together into batches. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the text samples in a batch concurrently."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "545PP3o8IrJV"
},
"source": [
"## Fine-tuning the model"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "FBiW8UpKIrJW"
},
"source": [
"Now that our data is ready for training, we can download the pre-trained model and fine-tune it. Since our task is question answering, we use the `AutoModelForQuestionAnswering` class. Like with the tokenizer, the `from_pretrained` method will download and cache the model for us:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TlqNaB8jIrJW",
"outputId": "84916cf3-6e6c-47f3-d081-032ec30a4132"
},
"outputs": [],
"source": [
"from transformers import AutoModelForQuestionAnswering\n",
"from optimum.graphcore import IPUConfig, IPUTrainer, IPUTrainingArguments\n",
"\n",
"model = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)\n",
"ipu_config = IPUConfig.from_pretrained(ipu_config_name, executable_cache_dir=executable_cache_dir)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "CczA5lJlIrJX"
},
"source": [
"The warning tells us we are throwing away some weights (the `vocab_transform` and `vocab_layer_norm` layers) and randomly initializing some others (the `pre_classifier` and `classifier` layers). This is normal in this case, because we are removing the head used to pre-train the model on a masked language modelling objective and replacing it with a new head for which we don't have pre-trained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "_N8urzhyIrJY"
},
"source": [
"To instantiate a `IPUTrainer` class, we will need to define two more things:\n",
"* `IPUTrainingArguments`, which is a class that contains all the attributes to customize the training. \n",
"* A data collator\n",
"\n",
"`IPUTrainingArguments` requires a folder name, which will be used to save the checkpoints of the model. All other arguments are optional:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Bliy8zgjIrJY"
},
"outputs": [],
"source": [
"model_name = model_checkpoint.split(\"/\")[-1]\n",
"args = IPUTrainingArguments(\n",
" f\"{model_name}-finetuned-squad\",\n",
" do_eval=True,\n",
" prediction_loss_only=False,\n",
" learning_rate=2e-5,\n",
" loss_scaling=64.0,\n",
" weight_decay=0.01,\n",
" warmup_ratio=0.25,\n",
" per_device_train_batch_size=micro_batch_size,\n",
" per_device_eval_batch_size=micro_batch_size,\n",
" n_ipu=n_ipu,\n",
" gradient_accumulation_steps=gradient_accumulation_steps,\n",
" num_train_epochs=3,\n",
" logging_steps=20,\n",
" pad_on_batch_axis=True,\n",
" push_to_hub=False,\n",
" # hub_model_id = f\"username-or-organization/{model_name}-finetuned-squad\"\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "km3pGVdTIrJc"
},
"source": [
"Here we set the evaluation to be done at the end of each epoch, tweak the learning rate and use the gloal batch size defined at the top of the notebook through `micro_batch_size`, `gradient_accumulation_steps` and `n_ipu`. We also customize the number of epochs for training, as well as the weight decay.\n",
"\n",
"`push_to_hub` in `IPUTrainingArguments` is necessary if we want to push the model to the [🤗 Models Hub](https://huggingface.co/models) regularly during training. You can remove them if you didn't follow the installation steps at the beginning of this notebook. If you want to save your model locally to a name that is different to the name of the repository it will be pushed to, or if you want to push your model under an organization and not your name space, use the `hub_model_id` argument to set the repo name (it needs to be the full name, including your namespace: for instance `\"sgugger/roberta-finetuned-squad\"` or `\"huggingface/roberta-finetuned-squad\"`)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we will need a data collator that will batch our processed samples together. Here we will use the default data collator:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from transformers import default_data_collator\n",
"\n",
"data_collator = default_data_collator"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "rXuFTAzDIrJe"
},
"source": [
"We will evaluate our model and compute metrics in the next section (this is a very long operation, so we will only compute the evaluation loss during training).\n",
"\n",
"Then we just need to pass all of this along with our datasets to the `IPUTrainer` class:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "imY1oC3SIrJf"
},
"outputs": [],
"source": [
"trainer = IPUTrainer(\n",
" model,\n",
" ipu_config,\n",
" args,\n",
" train_dataset=tokenized_datasets[\"train\"],\n",
" eval_dataset=tokenized_datasets[\"validation\"],\n",
" data_collator=data_collator,\n",
" tokenizer=tokenizer,\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "CdzABDVcIrJg"
},
"source": [
"We can now fine-tune our model by calling the `train` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "uNx5pyRlIrJh",
"outputId": "077e661e-d36c-469b-89b8-7ff7f73541ec"
},
"outputs": [],
"source": [
"trainer.train()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Since this training is particularly long, let's save the model just in case we need to restart."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"trainer.save_model(\"test-squad-trained\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Evaluation"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Evaluating our model will require a bit more work, as we will need to map the predictions of our model to parts of the context. The model itself predicts logits for the start and end positions of our answers. If we take a batch from our validation dataloader, here is the output our model gives us:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"\n",
"for batch in trainer.get_eval_dataloader():\n",
" break\n",
"batch = {k: v.to(trainer.args.device) for k, v in batch.items()}\n",
"\n",
"with torch.no_grad():\n",
" output = model(**batch)\n",
"output.keys()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The output of the model is a dict-like object that contains the loss (since we provided labels), the start and end logits. We won't need the loss for our predictions, so let's have a look a the logits:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"output.start_logits.shape, output.end_logits.shape"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We have one logit for each feature and each token. The most obvious thing to predict an answer for each feature is to take the index for the maximum of the start logits as a start position and the index of the maximum of the end logits as an end position."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"output.start_logits.argmax(dim=-1), output.end_logits.argmax(dim=-1)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This will work great in a lot of cases, but what if this prediction gives us something impossible: the start position could be greater than the end position, or point to a span of text in the question instead of the answer. In that case, we might want to look at the second-best prediction to see if it gives a possible answer and select that instead.\n",
"\n",
"However, picking the second-best answer is not as easy as picking the best one. Is it the second-best index in the start logits with the best index in the end logits? Or is it the best index in the start logits with the second best index in the end logits? And if that second-best answer is not possible either, it gets even trickier for the third-best answer.\n",
"\n",
"\n",
"To classify our answers, we will use the score obtained by adding the start and end logits. We won't try to order all the possible answers and so we limit ourselves with a hyper-parameter we call `n_best_size`. We'll pick the best indices in the start and end logits and gather all the answers this predicts. After checking if each one is valid, we will sort them by their score and keep the best one. Here is how we would do this on the first feature in the batch:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"n_best_size = 20"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"start_logits = output.start_logits[0].cpu().numpy()\n",
"end_logits = output.end_logits[0].cpu().numpy()\n",
"# Gather the indices the best start/end logits:\n",
"start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"valid_answers = []\n",
"for start_index in start_indexes:\n",
" for end_index in end_indexes:\n",
" if start_index <= end_index: # We need to refine that test to check the answer is inside the context\n",
" valid_answers.append(\n",
" {\n",
" \"score\": start_logits[start_index] + end_logits[end_index],\n",
" \"text\": \"\" # We need to find a way to get back the original substring corresponding to the answer in the context\n",
" }\n",
" )"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"And then we can sort the `valid_answers` according to their `score` and only keep the best one. The only point left is how to check that a given span is inside the context (and not the question) and how to get back the text inside. To do this, we need to add two things to our validation features:\n",
"- The ID of the sample that generated the feature (since each sample can generate several features, as seen before).\n",
"- The offset mapping that will give us a map from the token indices to the character positions in the context.\n",
"\n",
"That's why we will re-process the validation set with the following function, which is slightly different from `prepare_train_features`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def prepare_validation_features(examples):\n",
" # Some of the questions have lots of whitespace on the left, which is not useful and will make the\n",
" # truncation of the context fail (the tokenized question will take a lots of space). So we remove that\n",
" # left whitespace\n",
" examples[\"question\"] = [q.lstrip() for q in examples[\"question\"]]\n",
"\n",
" # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results\n",
" # in one example possible giving several features when a context is long, each of those features having a\n",
" # context that overlaps a bit the context of the previous feature.\n",
" tokenized_examples = tokenizer(\n",
" examples[\"question\" if pad_on_right else \"context\"],\n",
" examples[\"context\" if pad_on_right else \"question\"],\n",
" truncation=\"only_second\" if pad_on_right else \"only_first\",\n",
" max_length=max_length,\n",
" stride=doc_stride,\n",
" return_overflowing_tokens=True,\n",
" return_offsets_mapping=True,\n",
" padding=\"max_length\",\n",
" )\n",
"\n",
" # Since one example might give us several features if it has a long context, we need a map from a feature to\n",
" # its corresponding example. This key gives us just that.\n",
" sample_mapping = tokenized_examples.pop(\"overflow_to_sample_mapping\")\n",
"\n",
" # We keep the example_id that gave us this feature and we will store the offset mappings.\n",
" tokenized_examples[\"example_id\"] = []\n",
"\n",
" for i in range(len(tokenized_examples[\"input_ids\"])):\n",
" # Grab the sequence corresponding to that example (to know what is the context and what is the question).\n",
" sequence_ids = tokenized_examples.sequence_ids(i)\n",
" context_index = 1 if pad_on_right else 0\n",
"\n",
" # One example can give several spans, this is the index of the example containing this span of text.\n",
" sample_index = sample_mapping[i]\n",
" tokenized_examples[\"example_id\"].append(examples[\"id\"][sample_index])\n",
"\n",
" # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token\n",
" # position is part of the context or not.\n",
" tokenized_examples[\"offset_mapping\"][i] = [\n",
" (o if sequence_ids[k] == context_index else None)\n",
" for k, o in enumerate(tokenized_examples[\"offset_mapping\"][i])\n",
" ]\n",
"\n",
" return tokenized_examples"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"And like before, we can apply this function to our validation set:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"validation_features = datasets[\"validation\"].map(\n",
" prepare_validation_features,\n",
" batched=True,\n",
" remove_columns=datasets[\"validation\"].column_names\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can get the predictions for all features by using the `IPUTrainer.predict` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"raw_predictions = trainer.predict(validation_features)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"`IPUTrainer` *hides* the columns that are not used by the model (here `example_id` and `offset_mapping` which we will need for our post-processing), so we unhide them back:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"validation_features.set_format(type=validation_features.format[\"type\"], columns=list(validation_features.features.keys()))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now refine the test we had before. Since we set `None` in the offset mappings when it corresponds to a part of the question, it's easy to check if an answer is fully inside the context. We also eliminate very long answers from our considerations (with a hyper-parameter we can tune)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"max_answer_length = 30"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"start_logits = output.start_logits[0].cpu().numpy()\n",
"end_logits = output.end_logits[0].cpu().numpy()\n",
"offset_mapping = validation_features[0][\"offset_mapping\"]\n",
"# The first feature comes from the first example. For the more general case, we will need to be match the example_id to\n",
"# an example index\n",
"context = datasets[\"validation\"][0][\"context\"]\n",
"\n",
"# Gather the indices the best start/end logits:\n",
"start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
"valid_answers = []\n",
"for start_index in start_indexes:\n",
" for end_index in end_indexes:\n",
" # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond\n",
" # to part of the input_ids that are not in the context.\n",
" if (\n",
" start_index >= len(offset_mapping)\n",
" or end_index >= len(offset_mapping)\n",
" or offset_mapping[start_index] is None\n",
" or offset_mapping[end_index] is None\n",
" or offset_mapping[start_index] == []\n",
" or offset_mapping[end_index] == []\n",
" ):\n",
" continue\n",
" # Don't consider answers with a length that is either < 0 or > max_answer_length.\n",
" if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
" continue\n",
" if start_index <= end_index: # We need to refine that test to check the answer is inside the context\n",
" start_char = offset_mapping[start_index][0]\n",
" end_char = offset_mapping[end_index][1]\n",
" valid_answers.append(\n",
" {\n",
" \"score\": start_logits[start_index] + end_logits[end_index],\n",
" \"text\": context[start_char: end_char]\n",
" }\n",
" )\n",
"\n",
"valid_answers = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[:n_best_size]\n",
"valid_answers"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We can compare to the actual ground-truth answer:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"datasets[\"validation\"][0][\"answers\"]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Our model picked the right one as the most likely answer!\n",
"\n",
"As we mentioned in the code above, this was easy on the first feature because we knew it comes from the first sample. For the other features, we will need a map between samples and their corresponding features. Also, since one sample can give several features, we will need to gather together all the answers in all the features generated by a given sample, then pick the best one. The following code builds a map from sample index to its corresponding features indices:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import collections\n",
"\n",
"examples = datasets[\"validation\"]\n",
"features = validation_features\n",
"\n",
"example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
"features_per_example = collections.defaultdict(list)\n",
"for i, feature in enumerate(features):\n",
" features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We're almost ready for our post-processing function. The last bit to deal with is the impossible answer (when `squad_v2 = True`). The code above only keeps answers that are inside the context, and we also need to get the score for the impossible answer (which has start and end indices corresponding to the index of the CLS token). When one example gives several features, we have to predict the impossible answer when all the features give a high score to the impossible answer (since one feature could predict the impossible answer just because the answer isn't in the part of the context it has access too), which is why the score of the impossible answer for one sample is the *minimum* of the scores for the impossible answer in each feature generated by the sample.\n",
"\n",
"We then predict the impossible answer when that score is greater than the score of the best non-impossible answer. All combined together, this gives us this post-processing function:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tqdm.auto import tqdm\n",
"\n",
"def postprocess_qa_predictions(examples, features, raw_predictions, n_best_size = 20, max_answer_length = 30):\n",
" all_start_logits, all_end_logits = raw_predictions\n",
" # Build a map example to its corresponding features.\n",
" example_id_to_index = {k: i for i, k in enumerate(examples[\"id\"])}\n",
" features_per_example = collections.defaultdict(list)\n",
" for i, feature in enumerate(features):\n",
" features_per_example[example_id_to_index[feature[\"example_id\"]]].append(i)\n",
"\n",
" # The dictionaries we have to fill.\n",
" predictions = collections.OrderedDict()\n",
"\n",
" # Logging.\n",
" print(f\"Post-processing {len(examples)} example predictions split into {len(features)} features.\")\n",
"\n",
" # Let's loop over all the examples!\n",
" for example_index, example in enumerate(tqdm(examples)):\n",
" # Those are the indices of the features associated to the current example.\n",
" feature_indices = features_per_example[example_index]\n",
"\n",
" min_null_score = None # Only used if squad_v2 is True.\n",
" valid_answers = []\n",
" \n",
" context = example[\"context\"]\n",
" # Looping through all the features associated to the current example.\n",
" for feature_index in feature_indices:\n",
" # We grab the predictions of the model for this feature.\n",
" start_logits = all_start_logits[feature_index]\n",
" end_logits = all_end_logits[feature_index]\n",
" # This is what will allow us to map some the positions in our logits to span of texts in the original\n",
" # context.\n",
" offset_mapping = features[feature_index][\"offset_mapping\"]\n",
"\n",
" # Update minimum null prediction.\n",
" cls_index = features[feature_index][\"input_ids\"].index(tokenizer.cls_token_id)\n",
" feature_null_score = start_logits[cls_index] + end_logits[cls_index]\n",
" if min_null_score is None or min_null_score < feature_null_score:\n",
" min_null_score = feature_null_score\n",
"\n",
" # Go through all possibilities for the `n_best_size` greater start and end logits.\n",
" start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
" end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()\n",
" for start_index in start_indexes:\n",
" for end_index in end_indexes:\n",
" # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond\n",
" # to part of the input_ids that are not in the context.\n",
" if (\n",
" start_index >= len(offset_mapping)\n",
" or end_index >= len(offset_mapping)\n",
" or offset_mapping[start_index] is None\n",
" or offset_mapping[end_index] is None\n",
" or offset_mapping[start_index] == []\n",
" or offset_mapping[end_index] == []\n",
" ):\n",
" continue\n",
" # Don't consider answers with a length that is either < 0 or > max_answer_length.\n",
" if end_index < start_index or end_index - start_index + 1 > max_answer_length:\n",
" continue\n",
"\n",
" start_char = offset_mapping[start_index][0]\n",
" end_char = offset_mapping[end_index][1]\n",
" valid_answers.append(\n",
" {\n",
" \"score\": start_logits[start_index] + end_logits[end_index],\n",
" \"text\": context[start_char: end_char]\n",
" }\n",
" )\n",
" \n",
" if len(valid_answers) > 0:\n",
" best_answer = sorted(valid_answers, key=lambda x: x[\"score\"], reverse=True)[0]\n",
" else:\n",
" # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid\n",
" # failure.\n",
" best_answer = {\"text\": \"\", \"score\": 0.0}\n",
" \n",
" # Let's pick our final answer: the best one or the null answer (only for squad_v2)\n",
" if not squad_v2:\n",
" predictions[example[\"id\"]] = best_answer[\"text\"]\n",
" else:\n",
" answer = best_answer[\"text\"] if best_answer[\"score\"] > min_null_score else \"\"\n",
" predictions[example[\"id\"]] = answer\n",
"\n",
" return predictions"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"And we can apply our post-processing function to our raw predictions:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"final_predictions = postprocess_qa_predictions(datasets[\"validation\"], validation_features, raw_predictions.predictions)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we can load the metric from the datasets library."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"metric = load_metric(\"squad_v2\" if squad_v2 else \"squad\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Then we can call `compute` on it. We just need to format the predictions and labels a bit as it expects a list of dictionaries and not one big dictionary. In the case of `squad_v2`, we also have to set a `no_answer_probability` argument (which we set to 0.0 here as we have already set the answer to empty if we picked it)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if squad_v2:\n",
" formatted_predictions = [{\"id\": k, \"prediction_text\": v, \"no_answer_probability\": 0.0} for k, v in final_predictions.items()]\n",
"else:\n",
" formatted_predictions = [{\"id\": k, \"prediction_text\": v} for k, v in final_predictions.items()]\n",
"references = [{\"id\": ex[\"id\"], \"answers\": ex[\"answers\"]} for ex in datasets[\"validation\"]]\n",
"metric.compute(predictions=formatted_predictions, references=references)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can now upload the result of the training to the 🤗 Hub:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# trainer.push_to_hub()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also share this model and other users can load it with the identifier \"your-username/the-name-you-picked\" so for instance:\n",
"\n",
"```python\n",
"from transformers import AutoModelForQuestionAnswering\n",
"\n",
"model = AutoModelForQuestionAnswering.from_pretrained(\"sgugger/my-awesome-model\")\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"Try out the other [IPU-powered Jupyter Notebooks](https://www.graphcore.ai/ipu-jupyter-notebooks) to see how how IPUs perform on other tasks."
]
}
],
"metadata": {
"colab": {
"name": "Question Answering on SQUAD",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3.8.10 64-bit",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"vscode": {
"interpreter": {
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
}
}
},
"nbformat": 4,
"nbformat_minor": 1
}