notebooks/multiple_choice.ipynb (932 lines of code) (raw):
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "rEJBSTyZIrIb"
},
"source": [
"# Multiple choice on IPU using RoBERTa - Fine-tuning"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "kTCFado4IrIc"
},
"source": [
"In this notebook, we will see how to fine-tune one of the [🤗 Transformers](https://github.com/huggingface/transformers) models to a multiple choice task, which is the task of selecting the most plausible inputs in a given selection. We will use [SWAG](https://www.aclweb.org/anthology/D18-1009/) dataset but you can adapt the pre-processing parts of this notebook to any other multiple choice dataset you like. You can also use your own data. SWAG is a dataset about common-sense reasoning, where each sample describes a situation then proposes four options that could go after it. "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"| Domain | Tasks | Model | Datasets | Workflow | Number of IPUs | Execution time |\n",
"|---------|-------|-------|----------|----------|--------------|--------------|\n",
"| Natural language processing | Text classification | roberta-base | SWAG | Fine-tuning | 4 | 16min |"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"[](https://www.graphcore.ai/join-community)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Environment setup\n",
"\n",
"The best way to run this demo is on Paperspace Gradient's cloud IPUs because everything is already set up for you.\n",
"\n",
"[](https://ipu.dev/3lI3fzQ)\n",
"\n",
"To run the demo using other IPU hardware, you need to have the Poplar SDK enabled. Refer to the [Getting Started guide](https://docs.graphcore.ai/en/latest/getting-started.html#getting-started) for your system for details on how to enable the Poplar SDK. Also refer to the [Jupyter Quick Start guide](https://docs.graphcore.ai/projects/jupyter-notebook-quick-start/en/latest/index.html) for how to set up Jupyter to be able to run this notebook on a remote IPU machine."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Dependencies and configuration\n",
"\n",
"In order to improve usability and support for future users, Graphcore would like to collect information about the\n",
"applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n",
"\n",
"- User progression through the notebook\n",
"- Notebook details: number of cells, code being run and the output of the cells\n",
"- Environment details\n",
"\n",
"You can disable logging at any time by running `%unload_ext graphcore_cloud_tools.notebook_logging.gc_logger` from any cell."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Install the dependencies for this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install \"optimum-graphcore==0.7\"\n",
"%pip install graphcore-cloud-tools[logger]@git+https://github.com/graphcore/graphcore-cloud-tools\n",
"%load_ext graphcore_cloud_tools.notebook_logging.gc_logger"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"This notebook is built to run with any model checkpoint from the [🤗 Model Hub](https://huggingface.co/models) as long as that model has a version with a multiple choice head and is supported by Optimum Graphcore. The IPU config files of the supported models are available in Graphcore's [🤗 account](https://huggingface.co/Graphcore). You can also create your own IPU config file locally. \n",
"\n",
"In this notebook, we are using both data parallelism and pipeline parallelism (see the [tutorial on efficient data loading](https://github.com/graphcore/examples/tree/master/tutorials/pytorch/efficient_data_loading) for more information). Therefore the global batch size, which is the actual number of samples used for the weight update, is determined with three factors:\n",
"- `global_batch_size = micro_batch_size * gradient_accumulation_steps * replication_factor`\n",
"\n",
"The replication factor is determined by the type of IPU Pod, which will be used as a key to select the replication factor from a dictionary defined in the IPU config file. For example, the dictionary in the IPU config file [Graphcore/gpt2-small-ipu](https://huggingface.co/Graphcore/gpt2-small-ipu/blob/main/ipu_config.json) looks like this:\n",
"- `\"replication_factor\": {\"pod4\": 1, \"pod8\": 2, \"pod16\": 4, \"pod32\": 8, \"pod64\": 16, \"default\": 1}`\n",
"\n",
"Depending on your model and the IPU Pod you are using, you might need to adjust these three batch-size-related arguments."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zVvslsfMIrIh"
},
"outputs": [],
"source": [
"model_checkpoint = \"roberta-base\"\n",
"\n",
"ipu_config_name = \"Graphcore/roberta-base-ipu\"\n",
"micro_batch_size = 2\n",
"gradient_accumulation_steps = 16"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Values for machine size and cache directories can be configured through environment variables or directly in the notebook:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"n_ipu = int(os.getenv(\"NUM_AVAILABLE_IPU\", 4))\n",
"executable_cache_dir = os.getenv(\"POPLAR_EXECUTABLE_CACHE_DIR\", \"/tmp/exe_cache/\") + \"/multiple_choice\""
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Sharing your model with the community "
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can share your model with the 🤗 community. You do this by completing the following steps:\n",
"\n",
"1. Store your authentication token from the 🤗 website. [Sign up to 🤗](https://huggingface.co/join) if you haven't already.\n",
"2. Execute the following cell and input your username and password."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from huggingface_hub import notebook_login\n",
"\n",
"notebook_login()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Then you need to install Git-LFS to manage large files:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!apt install git-lfs"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "whPRbBNbIrIl"
},
"source": [
"## Loading the dataset"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "W7QYTpxXIrIl"
},
"source": [
"We will use the [🤗 Datasets](https://github.com/huggingface/datasets) library to download the data. This can be easily done with the function `load_dataset`. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "IreSlFmlIrIm"
},
"outputs": [],
"source": [
"from datasets import load_dataset, load_metric"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "CKx2zKs5IrIq"
},
"source": [
"`load_dataset` will cache the dataset to avoid downloading it again the next time you run this cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 270,
"referenced_widgets": [
"69caab03d6264fef9fc5649bffff5e20",
"3f74532faa86412293d90d3952f38c4a",
"50615aa59c7247c4804ca5cbc7945bd7",
"fe962391292a413ca55dc932c4279fa7",
"299f4b4c07654e53a25f8192bd1d7bbd",
"ad04ed1038154081bbb0c1444784dcc2",
"7c667ad22b5740d5a6319f1b1e3a8097",
"46c2b043c0f84806978784a45a4e203b",
"80e2943be35f46eeb24c8ab13faa6578",
"de5956b5008d4fdba807bae57509c393",
"931db1f7a42f4b46b7ff8c2e1262b994",
"6c1db72efff5476e842c1386fadbbdba",
"ccd2f37647c547abb4c719b75a26f2de",
"d30a66df5c0145e79693e09789d96b81",
"5fa26fc336274073abbd1d550542ee33",
"2b34de08115d49d285def9269a53f484",
"d426be871b424affb455aeb7db5e822e",
"160bf88485f44f5cb6eaeecba5e0901f",
"745c0d47d672477b9bb0dae77b926364",
"d22ab78269cd4ccfbcf70c707057c31b",
"d298eb19eeff453cba51c2804629d3f4",
"a7204ade36314c86907c562e0a2158b8",
"e35d42b2d352498ca3fc8530393786b2",
"75103f83538d44abada79b51a1cec09e",
"f6253931d90543e9b5fd0bb2d615f73a",
"051aa783ff9e47e28d1f9584043815f5",
"0984b2a14115454bbb009df71c1cf36f",
"8ab9dfce29854049912178941ef1b289",
"c9de740e007141958545e269372780a4",
"cbea68b25d6d4ba09b2ce0f27b1726d5",
"5781fc45cf8d486cb06ed68853b2c644",
"d2a92143a08a4951b55bab9bc0a6d0d3",
"a14c3e40e5254d61ba146f6ec88eae25",
"c4ffe6f624ce4e978a0d9b864544941a",
"1aca01c1d8c940dfadd3e7144bb35718",
"9fbbaae50e6743f2aa19342152398186",
"fea27ca6c9504fc896181bc1ff5730e5",
"940d00556cb849b3a689d56e274041c2",
"5cdf9ed939fb42d4bf77301c80b8afca",
"94b39ccfef0b4b08bf2fb61bb0a657c1",
"9a55087c85b74ea08b3e952ac1d73cbe",
"2361ab124daf47cc885ff61f2899b2af",
"1a65887eb37747ddb75dc4a40f7285f2",
"3c946e2260704e6c98593136bd32d921",
"50d325cdb9844f62a9ecc98e768cb5af",
"aa781f0cfe454e9da5b53b93e9baabd8",
"6bb68d3887ef43809eb23feb467f9723",
"7e29a8b952cf4f4ea42833c8bf55342f",
"dd5997d01d8947e4b1c211433969b89b",
"2ace4dc78e2f4f1492a181bcd63304e7",
"bbee008c2791443d8610371d1f16b62b",
"31b1c8a2e3334b72b45b083688c1a20c",
"7fb7c36adc624f7dbbcb4a831c1e4f63",
"0b7c8f1939074794b3d9221244b1344d",
"a71908883b064e1fbdddb547a8c41743",
"2f5223f26c8541fc87e91d2205c39995"
]
},
"id": "s_AY1ATSIrIq",
"outputId": "fd0578d1-8895-443d-b56f-5908de9f1b6b"
},
"outputs": [],
"source": [
"datasets = load_dataset(\"swag\", \"regular\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "RzfPtOMoIrIu"
},
"source": [
"The `dataset` object itself is [`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict), which contains one key for the training, validation and test set (with more keys for the mismatched validation and test sets in the special case of `mnli`)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GWiVUF0jIrIv",
"outputId": "35e3ea43-f397-4a54-c90c-f2cf8d36873e"
},
"outputs": [],
"source": [
"datasets"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "u3EtYfeHIrIz"
},
"source": [
"To access an actual element, you need to select a split (\"train\" in the example), and then specify an index:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "X6HrpprwIrIz",
"outputId": "d7670bc0-42e4-4c09-8a6a-5c018ded7d95"
},
"outputs": [],
"source": [
"datasets[\"train\"][0]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "WHUmphG3IrI3"
},
"source": [
"To get a sense of what the data looks like, the following function will show some samples picked randomly from the dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "i3j8APAoIrI3"
},
"outputs": [],
"source": [
"from datasets import ClassLabel\n",
"import random\n",
"import pandas as pd\n",
"from IPython.display import display, HTML\n",
"\n",
"def show_random_elements(dataset, num_examples=10):\n",
" assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
" picks = []\n",
" for _ in range(num_examples):\n",
" pick = random.randint(0, len(dataset)-1)\n",
" while pick in picks:\n",
" pick = random.randint(0, len(dataset)-1)\n",
" picks.append(pick)\n",
" \n",
" df = pd.DataFrame(dataset[picks])\n",
" for column, typ in dataset.features.items():\n",
" if isinstance(typ, ClassLabel):\n",
" df[column] = df[column].transform(lambda i: typ.names[i])\n",
" display(HTML(df.to_html()))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SZy5tRB_IrI7",
"outputId": "ba8f2124-e485-488f-8c0c-254f34f24f13"
},
"outputs": [],
"source": [
"show_random_elements(datasets[\"train\"])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"Each sample in the dataset has a context composed of a first sentence (in the field `sent1`) and an introduction to the second sentence (in the field `sent2`). Then four possible endings are given (in the fields `ending0`, `ending1`, `ending2` and `ending3`) and the model must pick the right one (indicated in the field `label`). The following function lets us visualize a given sample:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def show_one(example):\n",
" print(f\"Context: {example['sent1']}\")\n",
" print(f\" A - {example['sent2']} {example['ending0']}\")\n",
" print(f\" B - {example['sent2']} {example['ending1']}\")\n",
" print(f\" C - {example['sent2']} {example['ending2']}\")\n",
" print(f\" D - {example['sent2']} {example['ending3']}\")\n",
" print(f\"\\nGround truth: option {['A', 'B', 'C', 'D'][example['label']]}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"show_one(datasets[\"train\"][0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"show_one(datasets[\"train\"][15])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "n9qywopnIrJH"
},
"source": [
"## Preprocessing the data"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "YVx71GdAIrJH"
},
"source": [
"Before we can feed those text samples to our model, we need to preprocess them. This is done by a 🤗 Transformers `Tokenizer` which will tokenize the inputs (including converting the tokens to their corresponding IDs in the pre-trained vocabulary) and put it in a format the model expects, as well as generate the other inputs that the model requires.\n",
"\n",
"To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n",
"\n",
"- We get a tokenizer that corresponds to the model architecture we want to use.\n",
"- We download the vocabulary used when pre-training this specific checkpoint.\n",
"\n",
"That vocabulary will be cached, so it's not downloaded again the next time we run the cell."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eXNLu_-nIrJI"
},
"outputs": [],
"source": [
"from transformers import AutoTokenizer\n",
" \n",
"tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "Vl6IidfdIrJK"
},
"source": [
"We pass `use_fast=True` to the call above to use one of the fast tokenizers (backed by Rust) from the 🤗 Tokenizers library. These fast tokenizers are available for almost all models, but if you got an error with the previous call, remove that argument."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "rowT4iCLIrJK"
},
"source": [
"You can directly call this tokenizer on one sentence or a pair of sentences:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a5hBlsrHIrJL",
"outputId": "acdaa98a-a8cd-4a20-89b8-cc26437bbe90"
},
"outputs": [],
"source": [
"tokenizer(\"Hello, this one sentence!\", \"And this sentence goes with it.\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "qo_0B1M2IrJM"
},
"source": [
"Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here, but they are required by the model we will instantiate later. You can learn more about them in this [tutorial on preprocessing](https://huggingface.co/transformers/preprocessing.html).\n",
"\n",
"To preprocess our dataset, we will thus need the names of the columns containing the sentence(s). The following dictionary keeps track of the correspondence of task to column names:"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "2C0hcmp9IrJQ"
},
"source": [
"We can then write the function that will preprocess our samples. The tricky part is to put all the possible pairs of sentences into two lists before passing them to the tokenizer, then un-flatten the result so that each sample has four input IDs, attention masks and so on.\n",
"\n",
"We also define a maximum sequence length (`max_seq_length`) to either pad or truncate our samples such that each sample is the same size. Then, when calling the tokenizer, we use the arguments `truncation=True`, `padding=\"max_length\"` and `max_length=max_seq_length`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vc0BSBLIIrJQ"
},
"outputs": [],
"source": [
"ending_names = [\"ending0\", \"ending1\", \"ending2\", \"ending3\"]\n",
"max_seq_length = 128\n",
"\n",
"def preprocess_function(examples):\n",
" # Repeat each first sentence four times to go with the four possibilities of second sentences.\n",
" first_sentences = [[context] * 4 for context in examples[\"sent1\"]]\n",
" # Grab all second sentences possible for each context.\n",
" question_headers = examples[\"sent2\"]\n",
" second_sentences = [[f\"{header} {examples[end][i]}\" for end in ending_names] for i, header in enumerate(question_headers)]\n",
" \n",
" # Flatten everything\n",
" first_sentences = sum(first_sentences, [])\n",
" second_sentences = sum(second_sentences, [])\n",
" \n",
" # Tokenize\n",
" tokenized_examples = tokenizer(\n",
" first_sentences, \n",
" second_sentences, \n",
" truncation=True,\n",
" max_length=max_seq_length,\n",
" padding=\"max_length\",\n",
" )\n",
" # Un-flatten\n",
" return {k: [v[i:i+4] for i in range(0, len(v), 4)] for k, v in tokenized_examples.items()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "0lm8ozrJIrJR"
},
"source": [
"This function works with one or several samples. In the case of several samples, the tokenizer will return a list of lists of lists for each key: a list of all samples (here 5), then a list of all choices (4) and a list of input IDs:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"examples = datasets[\"train\"][:5]\n",
"features = preprocess_function(examples)\n",
"print(len(features[\"input_ids\"]), len(features[\"input_ids\"][0]), [len(x) for x in features[\"input_ids\"][0]])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"To check we didn't do anything wrong when grouping all possibilities then un-flattening, let's have a look at the decoded inputs for a given sample:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"idx = 3\n",
"[tokenizer.decode(features[\"input_ids\"][idx][i]) for i in range(4)]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We can compare it to the ground truth:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"show_one(datasets[\"train\"][3])"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "zS-6iXTkIrJT"
},
"source": [
"This seems right, so we can apply this function to all the samples in our dataset by using the `map` method of the `dataset` object we created earlier. This will apply the function to all the elements of all the splits in `dataset`. This means that our training, validation and testing data will be preprocessed with a single command."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "DDtsaJeVIrJT",
"outputId": "aa4734bf-4ef5-4437-9948-2c16363da719"
},
"outputs": [],
"source": [
"encoded_datasets = datasets.map(preprocess_function, batched=True)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "voWiw8C7IrJV"
},
"source": [
"Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library is able to detect when the function you pass to `map` has changed (and thus not to use the cached data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files and you can pass `load_from_cache_file=False` in the call to `map` to not use the cached files and to force the preprocessing to be applied again.\n",
"\n",
"Note that we passed `batched=True` to encode the text samples together into batches. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the text samples in a batch concurrently."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "545PP3o8IrJV"
},
"source": [
"## Fine-tuning the model"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "FBiW8UpKIrJW"
},
"source": [
"Now that our data is ready, we can download the pre-trained model and fine-tune it. Since all our tasks are multiple choice tasks, we use the `AutoModelForMultipleChoice` class. Like with the tokenizer, the `from_pretrained` method will download and cache the model for us. We also define the `IPUConfig` class, which is a class that specifies attributes and configuration parameters to compile and put the model on the device. We initialize it with a config name or a path, which we set earlier."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TlqNaB8jIrJW",
"outputId": "84916cf3-6e6c-47f3-d081-032ec30a4132"
},
"outputs": [],
"source": [
"from transformers import AutoModelForMultipleChoice\n",
"from optimum.graphcore import IPUConfig, IPUTrainer, IPUTrainingArguments\n",
"\n",
"model = AutoModelForMultipleChoice.from_pretrained(model_checkpoint)\n",
"ipu_config = IPUConfig.from_pretrained(ipu_config_name, executable_cache_dir=executable_cache_dir)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "CczA5lJlIrJX"
},
"source": [
"The warning tells us we are throwing away some weights (the `vocab_transform` and `vocab_layer_norm` layers) and randomly initializing some others (the `pre_classifier` and `classifier` layers). This is normal in this case, because we are removing the head used to pre-train the model on a masked language modelling objective and replacing it with a new head for which we don't have pre-trained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "_N8urzhyIrJY"
},
"source": [
"To instantiate the `IPUTrainer` class, we will need to define:\n",
"* `IPUTrainingArguments`, which is a class that contains all the attributes to customize the training.\n",
"* How to compute the metrics from the predictions\n",
"\n",
"`IPUTrainingArguments` requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Bliy8zgjIrJY"
},
"outputs": [],
"source": [
"model_name = model_checkpoint.split(\"/\")[-1]\n",
"args = IPUTrainingArguments(\n",
" f\"{model_name}-finetuned-swag\",\n",
" evaluation_strategy = \"epoch\",\n",
" learning_rate=5e-5,\n",
" per_device_train_batch_size=micro_batch_size,\n",
" per_device_eval_batch_size=micro_batch_size,\n",
" n_ipu=n_ipu,\n",
" gradient_accumulation_steps=gradient_accumulation_steps,\n",
" num_train_epochs=3,\n",
" weight_decay=0.01,\n",
" logging_steps=10,\n",
" dataloader_drop_last=True,\n",
" push_to_hub=False,\n",
" # hub_model_id = f\"username-or-organization/{model_name}-finetuned-squad\"\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "km3pGVdTIrJc"
},
"source": [
"Here we set the evaluation to be done at the end of each epoch, tweak the learning rate and use the global batch size defined earlier in the notebook through `micro_batch_size`, `gradient_accumulation_steps` and `n_ipu`. We also customize the number of epochs for training, as well as the weight decay.\n",
"\n",
"`push_to_hub` in `IPUTrainingArguments` is necessary if we want to push the model to the [🤗 Models Hub](https://huggingface.co/models) regularly during training. You can remove them if you didn't follow the installation steps at the beginning of this notebook. If you want to save your model locally to a name that is different to the name of the repository it will be pushed to, or if you want to push your model under an organization and not your name space, use the `hub_model_id` argument to set the repo name (it needs to be the full name, including your namespace: for instance `\"sgugger/bert-finetuned-swag\"` or `\"huggingface/bert-finetuned-swag\"`)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "7sZOdRlRIrJd"
},
"source": [
"The last thing to define for before we instantiate `IPUTrainer` is how to compute the metrics from the predictions. We need to define a function for this, which will use `metric`, which we loaded earlier. The only preprocessing we have to do is to calculate `argmax` for our predicted logits:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "UmvbnJ9JIrJd"
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"def compute_metrics(eval_predictions):\n",
" predictions, label_ids = eval_predictions\n",
" preds = np.argmax(predictions, axis=1)\n",
" return {\"accuracy\": (preds == label_ids).astype(np.float32).mean().item()}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "rXuFTAzDIrJe"
},
"source": [
"Then all we need to do is to pass all of this together with our datasets to `IPUTrainer`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "imY1oC3SIrJf"
},
"outputs": [],
"source": [
"from transformers import default_data_collator\n",
"\n",
"trainer = IPUTrainer(\n",
" model,\n",
" ipu_config,\n",
" args,\n",
" train_dataset=encoded_datasets[\"train\"],\n",
" eval_dataset=encoded_datasets[\"validation\"],\n",
" tokenizer=tokenizer,\n",
" data_collator=default_data_collator,\n",
" compute_metrics=compute_metrics,\n",
")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "CdzABDVcIrJg"
},
"source": [
"We can now fine-tune our model by calling the `train` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"trainer.train()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"We can check how our model did with the `evaluate` method:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"trainer.evaluate()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "wY82caEX3l_i"
},
"source": [
"You can now upload the result of the training to the 🤗 Hub:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# trainer.push_to_hub()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also share this model and other users can load it with the identifier \"your-username/the-name-you-picked\" so for instance:\n",
"\n",
"\n",
"```python\n",
"from transformers import AutoModelForMultipleChoice\n",
"\n",
"model = AutoModelForMultipleChoice.from_pretrained(\"sgugger/my-awesome-model\")\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"Try out the other [IPU-powered Jupyter Notebooks](https://www.graphcore.ai/ipu-jupyter-notebooks) to see how how IPUs perform on other tasks."
]
}
],
"metadata": {
"colab": {
"name": "Multiple choice on SWAG",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3.8.10 64-bit",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.10"
},
"vscode": {
"interpreter": {
"hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
}
}
},
"nbformat": 4,
"nbformat_minor": 1
}