sdk/python/jobs/automl-standalone-jobs/automl-nlp-text-named-entity-recognition-task-distributed-sweeping/automl-nlp-text-ner-task-distributed-with-sweeping.ipynb

{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# AutoML: Train \"the best\" NLP NER model for the CoNLL 2003 dataset.\n", "\n", "**Requirements** - In order to benefit from this tutorial, you will need:\n", "- A basic understanding of Machine Learning\n", "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n", "- An Azure ML workspace. [Check this notebook for creating a workspace](../../../resources/workspace/workspace.ipynb) \n", "- A Compute Cluster. [Check this notebook to create a compute cluster](../../../resources/compute/compute.ipynb)\n", "- A python environment\n", "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../README.md) - check the getting started section\n", "- Installed azure-identity package\n", "\n", "\n", "**Learning Objectives** - By the end of this tutorial, you should be able to:\n", "- Connect to your AML workspace from the Python SDK\n", "- Create an `AutoML Text Named Entity Recognition Training Job` with the 'text_ner()' factory-function\n", "- Specify custom models and hyperparameters to sweep over during training ***(Public Preview)*** \n", "- Leverage multi-node distribution to accelerate large model training\n", "- Obtain the model and score predictions with it\n", "\n", "Named entity recognition (NER) is a sub-task of information extraction (IE) that seeks out and categorizes specified entities in a body or bodies of texts. NER is also known simply as entity identification, entity chunking and entity extraction.\n", "\n", "This notebook trains a model using prepared datasets derived from the CoNLL-2003 dataset, introduced by Sang et al. in [Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition](https://paperswithcode.com/paper/introduction-to-the-conll-2003-shared-task). The derived version is available on KAGGLE: [CoNLL003 (English-version)](https://www.kaggle.com/datasets/alaakhaled/conll003-englishversion?select=valid.txt). Below, we go over how you can use AutoML for training a Text NER model. We will use the CoNLL dataset to train, demonstrate how you can sweep over models to get the best-performing one for the task at hand, and deploy the model to use in inference scenarios." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# 1. Connect to Azure Machine Learning Workspace\n", "\n", "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n", "\n", "## 1.1. Import the required libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "gather": { "logged": 1634852261599 } }, "outputs": [], "source": [ "# Import required libraries\n", "from azure.identity import DefaultAzureCredential\n", "from azure.ai.ml import MLClient\n", "\n", "from azure.ai.ml import Input\n", "from azure.ai.ml.constants import AssetTypes, NlpLearningRateScheduler\n", "from azure.ai.ml.automl import SearchSpace\n", "from azure.ai.ml.sweep import Choice, Uniform, BanditPolicy\n", "\n", "from azure.ai.ml import automl" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 1.2. Configure workspace details and get a handle to the workspace\n", "\n", "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../configuration.ipynb) for more details on how to configure credentials and connect to a workspace." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "gather": { "logged": 1634852261884 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } } }, "outputs": [], "source": [ "credential = DefaultAzureCredential()\n", "ml_client = None\n", "try:\n", " ml_client = MLClient.from_config(credential)\n", "except Exception as ex:\n", " print(ex)\n", " # Enter details of your AML workspace\n", " subscription_id = \"<SUBSCRIPTION_ID>\"\n", " resource_group = \"<RESOURCE_GROUP>\"\n", " workspace = \"<AML_WORKSPACE_NAME>\"\n", " ml_client = MLClient(credential, subscription_id, resource_group, workspace)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Show Azure ML Workspace information" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "workspace = ml_client.workspaces.get(name=ml_client.workspace_name)\n", "\n", "output = {}\n", "output[\"Workspace\"] = ml_client.workspace_name\n", "output[\"Subscription ID\"] = ml_client.connections._subscription_id\n", "output[\"Resource Group\"] = workspace.resource_group\n", "output[\"Location\"] = workspace.location\n", "output" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# 2. Data\n", "\n", "This model training uses the datasets from KAGGLE [CoNLL003 (English-version)](https://www.kaggle.com/datasets/alaakhaled/conll003-englishversion?select=valid.txt), in particular using the following datasets in the training and validation process:\n", "\n", "- Training dataset file (train.txt)\n", "- Validation dataset file (valid.txt)\n", "\n", "Both files are placed within their related MLTable folder.\n", "\n", "Please make use of the MLTable files present in separate folders at the same location (in the repo) as this notebook." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# MLTable folders\n", "training_mltable_path = \"./training-mltable-folder/\"\n", "validation_mltable_path = \"./validation-mltable-folder/\"\n", "\n", "# Training MLTable defined locally, with local data to be uploaded\n", "my_training_data_input = Input(type=AssetTypes.MLTABLE, path=training_mltable_path)\n", "\n", "# Validation MLTable defined locally, with local data to be uploaded\n", "my_validation_data_input = Input(type=AssetTypes.MLTABLE, path=validation_mltable_path)\n", "\n", "# WITH REMOTE PATH: If available already in the cloud/workspace-blob-store\n", "# my_training_data_input = Input(type=AssetTypes.MLTABLE, path=\"azureml://datastores/workspaceblobstore/paths/my_training_mltable\")\n", "# my_validation_data_input = Input(type=AssetTypes.MLTABLE, path=\"azureml://datastores/workspaceblobstore/paths/my_validation_mltable\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "For documentation on creating your own MLTable assets for jobs beyond this notebook:\n", "- https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-mltable details how to write MLTable YAMLs (required for each MLTable asset).\n", "- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-data-assets?tabs=Python-SDK covers how to work with them in the v2 CLI/SDK." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# 3. Compute target setup\n", "\n", "You will need to provide a [Compute Target](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#computes) that will be used for your AutoML model training. AutoML models for NLP tasks require [GPU SKUs](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu) such as the ones from the NC, NCv2, NCv3, ND, NDv2 and NCasT4 series. We recommend using the NCsv3-series (with v100 GPUs) for faster training. Using a compute target with a multi-GPU VM SKU will leverage the multiple GPUs to speed up training. Additionally, setting up a compute target with multiple nodes will allow for faster training, either by leveraging parallelism when exploring the model search space, or by distributing per-model training across multiple nodes." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from azure.ai.ml.entities import AmlCompute\n", "from azure.core.exceptions import ResourceNotFoundError\n", "\n", "compute_name = \"gpu-cluster-nc6s-v3\"\n", "\n", "try:\n", " _ = ml_client.compute.get(compute_name)\n", " print(\"Found existing compute target.\")\n", "except ResourceNotFoundError:\n", " print(\"Creating a new compute target...\")\n", " compute_config = AmlCompute(\n", " name=compute_name,\n", " type=\"amlcompute\",\n", " size=\"Standard_NC6s_v3\",\n", " idle_time_before_scale_down=120,\n", " min_instances=0,\n", " max_instances=4,\n", " )\n", " ml_client.begin_create_or_update(compute_config).result()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Configure and run the AutoML NLP Text NER training job\n", "AutoML allows you to easily train models for Text Classification (single- or multi-label) and Named Entity Recognition on your text data. You can control the model algorithm to be used, specify hyperparameter values for your model, as well as perform a sweep across the hyperparameter space to generate an optimal model.\n", "\n", "When using AutoML for text tasks, you can specify the model algorithm using the `model_name` parameter. You can either specify a single model or choose to sweep over multiple models. Please refer to the <a href=\"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-nlp-models?tabs=python#model-sweeping-and-hyperparameter-tuning-preview\">docs</a> for the list of supported models and hyperparameters." ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 4.1 Train with default hyperparameters for a single, specified algorithm\n", "Before doing a large sweep to search for the optimal models and hyperparameters, we recommend trying the default values for a given model to get a first baseline. Next, you can explore different models and hyperparameters, allowing for an iterative approach. With multiple models and hyperparameters, the search space grows exponentially, meaning you will need more iterations to find optimal configurations.\n", "\n", "The following funtions are used to configure the AutoML NLP job:\n", "\n", "### text_ner() function parameters:\n", "The `text_ner()` factory function allows the user to configure the training job.\n", "- `compute` - the compute on which the AutoML job will run. In this example we are using a compute called 'gpu-cluster' present in the workspace. You can replace it with any other compute in the workspace.\n", "- `experiment_name` - the name of the experiment. An experiment is like a folder with multiple runs from the AzureML Workspace that should be related to the same logical machine learning experiment.\n", "- `name` - the name of the Job/Run. This is an optional property. If not specified, a random name will be generated.\n", "- `primary_metric` - the metric that AutoML will optimize for during sweeping.\n", "\n", "### set_limits() function parameters:\n", "This is an optional configuration method to set limit parameters such as timeouts.\n", "- `timeout_minutes` - maximum amount of time in minutes that the whole AutoML job can take before the job terminates. If not specified, the default job's total timeout is 6 days (8,640 minutes).\n", "- `max_nodes` - if the underlying compute is a multi-node cluster, specify the maximum number of nodes to use for the experiment. The default is 1. This value can be increased to enable multi-node distribution. Note that if insufficient nodes are available on the compute compared to this value, a smaller value is used.\n", "\n", "### set_training_parameters() function parameters:\n", "This is an optional configuration method ***(public preview)*** to configure fixed settings or parameters that will _not_ be changed during the job parameter space sweeping. Specifying a `model_name` for instance fixes that model during training, and a range of models should not be specified in the parameter sweeping space for that same job. Some key parameters of this function are:\n", "- `model_name` - the name of the ML algorithm, or model, that we want to use during training.\n", "- `learning_rate` - the initial learning rate to use during training.\n", "- `learning_rate_scheduler` - the learning rate scheduler to use during training.\n", "- `warmup_ratio` - ratio of total training steps used to warmup from 0 to the initial `learning_rate`.\n", "\n", "Please refer to <a href=\"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-nlp-models?tabs=python#model-sweeping-and-hyperparameter-tuning-preview\">docs</a> for the full list of supported NLP models and hyperparameters.\n", " \n", "Now for an example, if you wish to run 2-way distributed training for a given model algorithm, say `roberta-base`, with a linear learning rate warmup of 10% of the total training steps, you can specify the job for your AutoML NLP runs as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [ "parameters" ] }, "outputs": [], "source": [ "# general job parameters\n", "exp_name = \"dpv2-nlp-text-ner-experiment\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "gather": { "logged": 1634852262026 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "name": "text-ner-configuration", "nteract": { "transient": { "deleting": false } }, "scrolled": true }, "outputs": [], "source": [ "# Create the AutoML job with the related factory-function.\n", "\n", "text_ner_job = automl.text_ner(\n", " compute=compute_name,\n", " # name=\"dpv2-text-ner-job-01\",\n", " experiment_name=exp_name,\n", " training_data=my_training_data_input,\n", " validation_data=my_validation_data_input,\n", " tags={\"my_custom_tag\": \"My custom value\"},\n", ")\n", "\n", "\n", "# Set limits\n", "text_ner_job.set_limits(timeout_minutes=120, max_nodes=2)\n", "\n", "# Pass the fixed parameters\n", "text_ner_job.set_training_parameters(\n", " model_name=\"roberta-base\",\n", " learning_rate_scheduler=NlpLearningRateScheduler.LINEAR,\n", " warmup_ratio=0.1,\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Submitting an AutoML job for NLP tasks\n", "Once you've configured the job, you can submit it in the workspace in order to train an NLP model using your training dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "gather": { "logged": 1634852267930 }, "jupyter": { "outputs_hidden": false, "source_hidden": false }, "nteract": { "transient": { "deleting": false } }, "scrolled": true }, "outputs": [], "source": [ "# Submit the AutoML job\n", "returned_job = ml_client.jobs.create_or_update(\n", " text_ner_job\n", ") # submit the job to the backend\n", "\n", "print(f\"Created job: {returned_job}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_client.jobs.stream(returned_job.name)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 4.2 Model & Hyperparameter Sweeping for AutoML NLP (Public Preview)\n", "When using AutoML NLP, we can perform a _sweep_ over a defined parameter space to find the optimal model and hyperparameters. Note that generally, for large pretrained text DNNs, hyperparameter sweeping often leads to less lift than switching to a more powerful model, so we focus our sweeping search space on model exploration. Whenever hyperparameters are not specified, default values are used for the specified algorithm.\n", "\n", "### set_limits() parameters\n", "The `set_limits` function has some useful limits specific to sweep procedures:\n", "- `max_trials` - parameter for maximum number of configurations to sweep. This must be an integer between 1 and 1000. Defaults to 1.\n", "- `max_concurrent_trials` - maximum number of runs that can run concurrently. If not specified, defaults to 1. If specified, the value must be an integer between 1 and 100. **Note**: if `max_nodes` is also specified, concurrent scheduling is given priority over multi-node distribution. For example, given an 8 node cluster with `max_nodes=4` and `max_concurrent_trials=4`, four single-node runs will be scheduled at all times until the max_trials limit is exhausted. With `max_nodes=8` and `max_concurrent_trials=4`, only then would we see four two-node distributed runs active at all times.\n", "\n", "\n", "### set_sweep() parameters\n", "The `set_sweep` function is used to configure the sweep settings:\n", "- `sampling_algorithm` - sampling method to use for sweeping over the defined parameter space. Please refer to <a href=\"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-nlp-models?tabs=cli#sampling-methods-for-the-sweep\">docs</a> for the list of supported sampling methods.\n", "- `early_termination` - early termination policy to end poorly performing runs. If no termination policy is specified, all configurations are run to completion. Please refer to this <a href=\"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters#early-termination\">page</a> for supported early termination policies.\n", "\n", "In the following example, we use random sampling to pick samples from the parameter space and specify a total of 4 iterations, running 2 iterations at a time on our compute target.\n", " \n", "We leverage the Bandit early termination policy, which will terminate poorly performing configs (those that are not within 5% slack of the best performing config), thus significantly saving compute resources.\n", " \n", "For more details on model and hyperparameter sweeping, please refer to the <a href=\"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-nlp-models?tabs=cli#model-sweeping-and-hyperparameter-tuning-preview\">docs</a>." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create the AutoML job with the related factory-function.\n", "\n", "text_ner_job = automl.text_ner(\n", " compute=compute_name,\n", " # name=\"dpv2-text-ner-job-02\",\n", " experiment_name=exp_name,\n", " training_data=my_training_data_input,\n", " validation_data=my_validation_data_input,\n", " tags={\"my_custom_tag\": \"My custom value\"},\n", ")\n", "\n", "text_ner_job.set_limits(\n", " timeout_minutes=120, max_trials=4, max_concurrent_trials=2, max_nodes=4\n", ")\n", "\n", "text_ner_job.extend_search_space(\n", " [\n", " SearchSpace(\n", " model_name=Choice([\"bert-base-cased\", \"roberta-base\"]),\n", " ),\n", " SearchSpace(\n", " model_name=Choice([\"distilroberta-base\"]),\n", " weight_decay=Uniform(0.01, 0.1),\n", " ),\n", " ]\n", ")\n", "\n", "text_ner_job.set_sweep(\n", " sampling_algorithm=\"Random\",\n", " early_termination=BanditPolicy(\n", " evaluation_interval=2, slack_factor=0.05, delay_evaluation=6\n", " ),\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Submit the AutoML job\n", "returned_job = ml_client.jobs.create_or_update(\n", " text_ner_job\n", ") # submit the job to the backend\n", "\n", "print(f\"Created job: {returned_job}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_client.jobs.stream(returned_job.name)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "When sweeping through the parameters in the provided search space, it can be useful to visualize the different configurations that were tried using the HyperDrive UI. You can navigate to this UI by going to the 'Child jobs' tab in the UI of the main automl nlp job from above, which is the HyperDrive parent run. Then you can go into the 'Trials' tab of this HyperDrive parent run. ALternatively, here below you can see directly the HyperDrive parent run and navigate to its 'Trials' tab:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hd_job = ml_client.jobs.get(returned_job.name + \"_HD\")\n", "hd_job" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 4.3 Manual hyperparameter sweeping for models from Hugging Face (Preview)\n", "You can use any model algorithm from Hugging face transformers library for either an individual run or you can also include these models to perform a hyperparameter sweep. You can also choose a combination of model algorithms supported supported natively by [AutoML](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-nlp-models?view=azureml-api-2&tabs=cli) and model algorithms from [Hugging Face](https://huggingface.co/models?pipeline_tag=token-classification&library=transformers&sort=trending).\n", "\n", "In this example, we sweep over bert-base-cased, microsoft/xdoc-base-funsd, and xlm-roberta-large-finetuned-conll03-english, models choosing from a range of values for learning_rate, number_of_epochs, etc., to generate a model with the optimal 'accuracy'." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create the AutoML job with the related factory-function.\n", "\n", "text_ner_job = automl.text_ner(\n", " compute=compute_name,\n", " # name=\"dpv2-text-ner-job-02\",\n", " experiment_name=exp_name,\n", " training_data=my_training_data_input,\n", " validation_data=my_validation_data_input,\n", " tags={\"my_custom_tag\": \"My custom value\"},\n", ")\n", "\n", "text_ner_job.set_limits(timeout_minutes=120, max_trials=4, max_concurrent_trials=2)\n", "\n", "text_ner_job.extend_search_space(\n", " [\n", " SearchSpace(\n", " model_name=Choice([\"bert-large-cased\", \"roberta-base\"]),\n", " ),\n", " SearchSpace(\n", " model_name=Choice([\"roberta-base-openai-detector\"]),\n", " weight_decay=Uniform(0.01, 0.1),\n", " ),\n", " ]\n", ")\n", "\n", "text_ner_job.set_sweep(\n", " sampling_algorithm=\"Random\",\n", " early_termination=BanditPolicy(\n", " evaluation_interval=2, slack_factor=0.05, delay_evaluation=6\n", " ),\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# 5. Retrieve the Best Model\n", "Once all the trials complete training, we can retrieve the best model and deploy it.\n", "\n", "## Initialize MLflow Client\n", "The models and artifacts that are produced by AutoML can be accessed via the MLflow interface. Initialize the MLflow client here and set the backend to Azure ML via the MLflow Client. IMPORTANT: you need to have installed the latest MLflow packages with:\n", "\n", "`pip install azureml-mlflow`\n", "\n", "`pip install mlflow`" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Obtain the tracking URI for MLflow" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import mlflow\n", "\n", "# Obtain the tracking URI from MLClient\n", "MLFLOW_TRACKING_URI = ml_client.workspaces.get(\n", " name=ml_client.workspace_name\n", ").mlflow_tracking_uri\n", "\n", "print(MLFLOW_TRACKING_URI)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Set the MLflow tracking URI\n", "mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)\n", "\n", "print(\"\\nCurrent tracking uri: {}\".format(mlflow.get_tracking_uri()))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from mlflow.tracking.client import MlflowClient\n", "from mlflow.artifacts import download_artifacts\n", "\n", "# Initialize MLflow client\n", "mlflow_client = MlflowClient()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Get the AutoML parent job" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "job_name = returned_job.name\n", "\n", "# Example if providing a specific job name/ID\n", "# job_name = \"joyful_carrot_rv9jrjk6c6\"\n", "\n", "# Get the parent run\n", "mlflow_parent_run = mlflow_client.get_run(job_name)\n", "\n", "print(\"Parent Run: \")\n", "print(mlflow_parent_run)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Print parent run tags. 'automl_best_child_run_id' tag should be there.\n", "print(mlflow_parent_run.data.tags)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Get the AutoML best child run" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Get the best model's child run\n", "best_child_run_id = mlflow_parent_run.data.tags[\"automl_best_child_run_id\"]\n", "print(\"Found best child run id: \", best_child_run_id)\n", "\n", "best_run = mlflow_client.get_run(best_child_run_id)\n", "\n", "print(\"Best child run: \")\n", "print(best_run)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Get best model run's metrics\n", "\n", "Access the results (such as Models, Artifacts, and Metrics) of a previously completed AutoML Run." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "pd.DataFrame(best_run.data.metrics, index=[0]).T" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Download the best model locally" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "# Create local folder\n", "local_dir = \"./artifact_downloads\"\n", "if not os.path.exists(local_dir):\n", " os.mkdir(local_dir)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Download run's artifacts/outputs\n", "local_path = download_artifacts(\n", " run_id=best_run.info.run_id, artifact_path=\"outputs\", dst_path=local_dir\n", ")\n", "print(\"Artifacts downloaded in: {}\".format(local_path))\n", "print(\"Artifacts: {}\".format(os.listdir(local_path)))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Show the contents of the MLflow model folder\n", "os.listdir(\"./artifact_downloads/outputs/mlflow-model\")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# 6. Register best model and deploy" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 6.1 Create managed online endpoint" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# import required libraries\n", "from azure.ai.ml.entities import (\n", " ManagedOnlineEndpoint,\n", " ManagedOnlineDeployment,\n", " Model,\n", " Environment,\n", " CodeConfiguration,\n", " ProbeSettings,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Creating a unique endpoint name with current datetime to avoid conflicts\n", "import datetime\n", "\n", "online_endpoint_name = \"conll-ner-\" + datetime.datetime.now().strftime(\"%m%d%H%M%f\")\n", "\n", "# create an online endpoint\n", "endpoint = ManagedOnlineEndpoint(\n", " name=online_endpoint_name,\n", " description=\"this is a sample online endpoint for deploying a model\",\n", " auth_mode=\"key\",\n", " tags={\"foo\": \"bar\"},\n", ")\n", "print(online_endpoint_name)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_client.begin_create_or_update(endpoint).result()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## 6.2 Deploy" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# deploying the mlflow-model\n", "model_name = \"conll-ner-mlflow-model\"\n", "model = Model(\n", " path=f\"azureml://jobs/{best_run.info.run_id}/outputs/artifacts/outputs/mlflow-model/\",\n", " name=model_name,\n", " description=\"my sample ner model\",\n", " type=AssetTypes.MLFLOW_MODEL,\n", ")\n", "\n", "# for downloaded file\n", "# model = Model(\n", "# path=path=\"artifact_downloads/outputs/mlflow-model/\",\n", "# name=model_name,\n", "# description=\"my sample instance segmentation model\",\n", "# type=AssetTypes.MLFLOW_MODEL,\n", "# )\n", "\n", "registered_model = ml_client.models.create_or_update(model)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "registered_model.id" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Deploy" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deployment = ManagedOnlineDeployment(\n", " name=\"conll-ner-mlflow-dpl\",\n", " endpoint_name=online_endpoint_name,\n", " model=registered_model.id,\n", " instance_type=\"Standard_DS4_V2\",\n", " instance_count=1,\n", " liveness_probe=ProbeSettings(\n", " failure_threshold=30,\n", " success_threshold=1,\n", " timeout=2,\n", " period=10,\n", " initial_delay=2000,\n", " ),\n", " readiness_probe=ProbeSettings(\n", " failure_threshold=10,\n", " success_threshold=1,\n", " timeout=10,\n", " period=10,\n", " initial_delay=2000,\n", " ),\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_client.online_deployments.begin_create_or_update(deployment).result()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# set our ner endpoint to take 100% of traffic\n", "endpoint.traffic = {\"conll-ner-mlflow-dpl\": 100}\n", "ml_client.begin_create_or_update(endpoint).result()" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Get endpoint details" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Get the details for online endpoint\n", "endpoint = ml_client.online_endpoints.get(name=online_endpoint_name)\n", "\n", "# existing traffic details\n", "print(endpoint.traffic)\n", "\n", "# Get the scoring URI\n", "print(endpoint.scoring_uri)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Test the deployment" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "CoNLL_formatted_string = \"\"\"The\n", "European\n", "Commission\n", "made\n", "a\n", "ruling\n", "on\n", "Friday\n", "\"\"\"\n", "request_json = {\"input_data\": [CoNLL_formatted_string]}" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import json\n", "\n", "request_file_name = \"sample_request_data.json\"\n", "with open(request_file_name, \"w\") as request_file:\n", " json.dump(request_json, request_file)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "resp = ml_client.online_endpoints.invoke(\n", " endpoint_name=online_endpoint_name,\n", " deployment_name=deployment.name,\n", " request_file=request_file_name,\n", ")" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "### Delete the deployment and endpoint\n", "\n", "Once you are done with the model, you can delete the endpoint and associated deployment if you wish." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml_client.online_endpoints.begin_delete(name=online_endpoint_name)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Next Steps" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "You can see further examples of other AutoML tasks, such as regression, image-classification, time-series forecasting, etc. in other notebooks of this repo." ] } ], "metadata": { "kernel_info": { "name": "python3-azureml" }, "kernelspec": { "display_name": "Python 3.10 - SDK V2", "language": "python", "name": "python310-sdkv2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" }, "microsoft": { "host": { "AzureML": { "notebookHasBeenCompleted": true } } }, "nteract": { "version": "nteract-front-end@1.0.0" } }, "nbformat": 4, "nbformat_minor": 4 }

sdk/python/jobs/automl-standalone-jobs/automl-nlp-text-named-entity-recognition-task-distributed-sweeping/automl-nlp-text-ner-task-distributed-with-sweeping.ipynb (1,025 lines of code) (raw):