transformers_doc/en/pytorch/image

{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Transformers installation\n", "! pip install transformers datasets evaluate accelerate\n", "# To install from source instead of the last release, comment the command above and uncomment the following one.\n", "# ! pip install git+https://github.com/huggingface/transformers.git" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Image classification" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "hide_input": true }, "outputs": [ { "data": { "text/html": [ "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/tjAIM7BOYhw?rel=0&controls=0&showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>" ], "text/plain": [ "<IPython.core.display.HTML object>" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#@title\n", "from IPython.display import HTML\n", "\n", "HTML('<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/tjAIM7BOYhw?rel=0&controls=0&showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Image classification assigns a label or class to an image. Unlike text or audio classification, the inputs are the\n", "pixel values that comprise an image. There are many applications for image classification, such as detecting damage\n", "after a natural disaster, monitoring crop health, or helping screen medical images for signs of disease.\n", "\n", "This guide illustrates how to:\n", "\n", "1. Fine-tune [ViT](https://huggingface.co/docs/transformers/main/en/tasks/../model_doc/vit) on the [Food-101](https://huggingface.co/datasets/food101) dataset to classify a food item in an image.\n", "2. Use your fine-tuned model for inference.\n", "\n", "<Tip>\n", "\n", "To see all architectures and checkpoints compatible with this task, we recommend checking the [task-page](https://huggingface.co/tasks/image-classification)\n", "\n", "</Tip>\n", "\n", "Before you begin, make sure you have all the necessary libraries installed:\n", "\n", "```bash\n", "pip install transformers datasets evaluate accelerate pillow torchvision scikit-learn\n", "```\n", "\n", "We encourage you to log in to your Hugging Face account to upload and share your model with the community. When prompted, enter your token to log in:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from huggingface_hub import notebook_login\n", "\n", "notebook_login()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load Food-101 dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Start by loading a smaller subset of the Food-101 dataset from the 🤗 Datasets library. This will give you a chance to\n", "experiment and make sure everything works before spending more time training on the full dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from datasets import load_dataset\n", "\n", "food = load_dataset(\"food101\", split=\"train[:5000]\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Split the dataset's `train` split into a train and test set with the [train_test_split](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.train_test_split) method:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "food = food.train_test_split(test_size=0.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then take a look at an example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=512x512 at 0x7F52AFC8AC50>,\n", " 'label': 79}" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "food[\"train\"][0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each example in the dataset has two fields:\n", "\n", "- `image`: a PIL image of the food item\n", "- `label`: the label class of the food item\n", "\n", "To make it easier for the model to get the label name from the label id, create a dictionary that maps the label name\n", "to an integer and vice versa:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "labels = food[\"train\"].features[\"label\"].names\n", "label2id, id2label = dict(), dict()\n", "for i, label in enumerate(labels):\n", " label2id[label] = str(i)\n", " id2label[str(i)] = label" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now you can convert the label id to a label name:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'prime_rib'" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "id2label[str(79)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preprocess" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The next step is to load a ViT image processor to process the image into a tensor:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from transformers import AutoImageProcessor\n", "\n", "checkpoint = \"google/vit-base-patch16-224-in21k\"\n", "image_processor = AutoImageProcessor.from_pretrained(checkpoint)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Apply some image transformations to the images to make the model more robust against overfitting. Here you'll use torchvision's [`transforms`](https://pytorch.org/vision/stable/transforms.html) module, but you can also use any image library you like.\n", "\n", "Crop a random part of the image, resize it, and normalize it with the image mean and standard deviation:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from torchvision.transforms import RandomResizedCrop, Compose, Normalize, ToTensor\n", "\n", "normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)\n", "size = (\n", " image_processor.size[\"shortest_edge\"]\n", " if \"shortest_edge\" in image_processor.size\n", " else (image_processor.size[\"height\"], image_processor.size[\"width\"])\n", ")\n", "_transforms = Compose([RandomResizedCrop(size), ToTensor(), normalize])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then create a preprocessing function to apply the transforms and return the `pixel_values` - the inputs to the model - of the image:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def transforms(examples):\n", " examples[\"pixel_values\"] = [_transforms(img.convert(\"RGB\")) for img in examples[\"image\"]]\n", " del examples[\"image\"]\n", " return examples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To apply the preprocessing function over the entire dataset, use 🤗 Datasets [with_transform](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.with_transform) method. The transforms are applied on the fly when you load an element of the dataset:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "food = food.with_transform(transforms)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now create a batch of examples using [DefaultDataCollator](https://huggingface.co/docs/transformers/main/en/main_classes/data_collator#transformers.DefaultDataCollator). Unlike other data collators in 🤗 Transformers, the `DefaultDataCollator` does not apply additional preprocessing such as padding." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from transformers import DefaultDataCollator\n", "\n", "data_collator = DefaultDataCollator()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluate" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Including a metric during training is often helpful for evaluating your model's performance. You can quickly load an\n", "evaluation method with the 🤗 [Evaluate](https://huggingface.co/docs/evaluate/index) library. For this task, load\n", "the [accuracy](https://huggingface.co/spaces/evaluate-metric/accuracy) metric (see the 🤗 Evaluate [quick tour](https://huggingface.co/docs/evaluate/a_quick_tour) to learn more about how to load and compute a metric):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import evaluate\n", "\n", "accuracy = evaluate.load(\"accuracy\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then create a function that passes your predictions and labels to [compute](https://huggingface.co/docs/evaluate/main/en/package_reference/main_classes#evaluate.EvaluationModule.compute) to calculate the accuracy:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "\n", "def compute_metrics(eval_pred):\n", " predictions, labels = eval_pred\n", " predictions = np.argmax(predictions, axis=1)\n", " return accuracy.compute(predictions=predictions, references=labels)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Your `compute_metrics` function is ready to go now, and you'll return to it when you set up your training." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "<Tip>\n", "\n", "If you aren't familiar with finetuning a model with the [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer), take a look at the basic tutorial [here](https://huggingface.co/docs/transformers/main/en/tasks/../training#train-with-pytorch-trainer)!\n", "\n", "</Tip>\n", "\n", "You're ready to start training your model now! Load ViT with [AutoModelForImageClassification](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForImageClassification). Specify the number of labels along with the number of expected labels, and the label mappings:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from transformers import AutoModelForImageClassification, TrainingArguments, Trainer\n", "\n", "model = AutoModelForImageClassification.from_pretrained(\n", " checkpoint,\n", " num_labels=len(labels),\n", " id2label=id2label,\n", " label2id=label2id,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "At this point, only three steps remain:\n", "\n", "1. Define your training hyperparameters in [TrainingArguments](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments). It is important you don't remove unused columns because that'll drop the `image` column. Without the `image` column, you can't create `pixel_values`. Set `remove_unused_columns=False` to prevent this behavior! The only other required parameter is `output_dir` which specifies where to save your model. You'll push this model to the Hub by setting `push_to_hub=True` (you need to be signed in to Hugging Face to upload your model). At the end of each epoch, the [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) will evaluate the accuracy and save the training checkpoint.\n", "2. Pass the training arguments to [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer) along with the model, dataset, tokenizer, data collator, and `compute_metrics` function.\n", "3. Call [train()](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.train) to finetune your model." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "training_args = TrainingArguments(\n", " output_dir=\"my_awesome_food_model\",\n", " remove_unused_columns=False,\n", " eval_strategy=\"epoch\",\n", " save_strategy=\"epoch\",\n", " learning_rate=5e-5,\n", " per_device_train_batch_size=16,\n", " gradient_accumulation_steps=4,\n", " per_device_eval_batch_size=16,\n", " num_train_epochs=3,\n", " warmup_ratio=0.1,\n", " logging_steps=10,\n", " load_best_model_at_end=True,\n", " metric_for_best_model=\"accuracy\",\n", " push_to_hub=True,\n", ")\n", "\n", "trainer = Trainer(\n", " model=model,\n", " args=training_args,\n", " data_collator=data_collator,\n", " train_dataset=food[\"train\"],\n", " eval_dataset=food[\"test\"],\n", " processing_class=image_processor,\n", " compute_metrics=compute_metrics,\n", ")\n", "\n", "trainer.train()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once training is completed, share your model to the Hub with the [push_to_hub()](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.push_to_hub) method so everyone can use your model:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "trainer.push_to_hub()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "<Tip>\n", "\n", "For a more in-depth example of how to finetune a model for image classification, take a look at the corresponding [PyTorch notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb).\n", "\n", "</Tip>" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Inference" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Great, now that you've fine-tuned a model, you can use it for inference!\n", "\n", "Load an image you'd like to run inference on:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ds = load_dataset(\"food101\", split=\"validation[:10]\")\n", "image = ds[\"image\"][0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "<div class=\"flex justify-center\">\n", " <img src=\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png\" alt=\"image of beignets\"/>\n", "</div>\n", "\n", "The simplest way to try out your finetuned model for inference is to use it in a [pipeline()](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.pipeline). Instantiate a `pipeline` for image classification with your model, and pass your image to it:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[{'score': 0.31856709718704224, 'label': 'beignets'},\n", " {'score': 0.015232225880026817, 'label': 'bruschetta'},\n", " {'score': 0.01519392803311348, 'label': 'chicken_wings'},\n", " {'score': 0.013022331520915031, 'label': 'pork_chop'},\n", " {'score': 0.012728818692266941, 'label': 'prime_rib'}]" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from transformers import pipeline\n", "\n", "classifier = pipeline(\"image-classification\", model=\"my_awesome_food_model\")\n", "classifier(image)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also manually replicate the results of the `pipeline` if you'd like:\n", "\n", "Load an image processor to preprocess the image and return the `input` as PyTorch tensors:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from transformers import AutoImageProcessor\n", "import torch\n", "\n", "image_processor = AutoImageProcessor.from_pretrained(\"my_awesome_food_model\")\n", "inputs = image_processor(image, return_tensors=\"pt\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pass your inputs to the model and return the logits:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from transformers import AutoModelForImageClassification\n", "\n", "model = AutoModelForImageClassification.from_pretrained(\"my_awesome_food_model\")\n", "with torch.no_grad():\n", " logits = model(**inputs).logits" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get the predicted label with the highest probability, and use the model's `id2label` mapping to convert it to a label:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'beignets'" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "predicted_label = logits.argmax(-1).item()\n", "model.config.id2label[predicted_label]" ] } ], "metadata": {}, "nbformat": 4, "nbformat_minor": 4 }

transformers_doc/en/pytorch/image_classification.ipynb (622 lines of code) (raw):