sdk/python/endpoints/online/deploy-with-packages/custom-model/sdk-deploy-and-test.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy models using packages to Online Endpoints\n",
    "\n",
    "In this examples, you will learn how to package model using the Model Packaging functionality to later deploy it in an online endpoint in Azure Machine Learning."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "Ensure you have the latest version of `azure-ai-ml`:\n",
    "\n",
    "```bash\n",
    "%pip install -U azure-ai-ml\n",
    "```"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
    "\n",
    "### 1.1. Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1682093988926
    }
   },
   "outputs": [],
   "source": [
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.ai.ml.entities import (\n",
    "    AzureMLOnlineInferencingServer,\n",
    "    ModelPackage,\n",
    "    CodeConfiguration,\n",
    "    BaseEnvironment,\n",
    "    ModelConfiguration,\n",
    ")\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    "    ManagedOnlineDeployment,\n",
    "    Environment,\n",
    "    Model,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Configure workspace details and get a handle to the workspace\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../jobs/configuration.ipynb) for more details on how to configure credentials and connect to a workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1682093991210
    }
   },
   "outputs": [],
   "source": [
    "subscription_id = \"<subscription>\"\n",
    "resource_group = \"<resource-group>\"\n",
    "workspace = \"<workspace>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are running on AzureML compute, you can easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1682093994327
    }
   },
   "outputs": [],
   "source": [
    "ml_client = MLClient.from_config(DefaultAzureCredential())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Register the model\n",
    "\n",
    "The model we will deploy is a Scikit-Learn model which is included in this repository in the folder `model`. We are going to register it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "register_model"
   },
   "outputs": [],
   "source": [
    "model_name = \"sklearn-regression\"\n",
    "model_path = \"model\"\n",
    "\n",
    "model = ml_client.models.create_or_update(Model(name=model_name, path=model_path))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Get the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "get_model"
   },
   "outputs": [],
   "source": [
    "model = ml_client.models.get(name=model_name, label=\"latest\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Create the endpoint\n",
    "\n",
    "Let's create an endpoint to host the deployment of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "name_endpoint"
   },
   "outputs": [],
   "source": [
    "endpoint_name = \"hello-packages\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Endpoint names should be unique so we will append a random string at the end to ensure that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1682094044091
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import string\n",
    "\n",
    "# Creating a unique endpoint name by including a random suffix\n",
    "allowed_chars = string.ascii_lowercase + string.digits\n",
    "endpoint_suffix = \"\".join(random.choice(allowed_chars) for x in range(5))\n",
    "endpoint_name = f\"{endpoint_name}-{endpoint_suffix}\"\n",
    "\n",
    "print(f\"Endpoint name: {endpoint_name}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Let's create the endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create_endpoint"
   },
   "outputs": [],
   "source": [
    "endpoint = ManagedOnlineEndpoint(name=endpoint_name)\n",
    "endpoint = ml_client.online_endpoints.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## 4. Deploy with packages"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 4.1 The base environment\n",
    "\n",
    "Packages uses a base environment to construct the package. The base environment contains all the software dependencies required for the model to run. Notice that, unlike when deploying models to Online Endpoints, this environment doesn't need any specific package for Online Endpoints. Only include the packages required for your model to work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "base_environment"
   },
   "outputs": [],
   "source": [
    "base_environment = ml_client.environments.create_or_update(\n",
    "    Environment(\n",
    "        name=f\"{model_name}-env\",\n",
    "        image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04\",\n",
    "        conda_file=\"environment/conda.yml\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 4.2 Package the model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "In our case, we will deploy to an online endpoint using the AzureML inferencing server, which is the default one. Since our model is not MLflow, we need to indicate a code configuration section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "configure_package"
   },
   "outputs": [],
   "source": [
    "pakage_config = ModelPackage(\n",
    "    target_environment=\"sklearn-regression-online-pkg\",\n",
    "    base_environment_source=BaseEnvironment(\n",
    "        type=\"asset\",\n",
    "        resource_id=f\"azureml:{base_environment.name}:{base_environment.version}\",\n",
    "    ),\n",
    "    inferencing_server=AzureMLOnlineInferencingServer(\n",
    "        code_configuration=CodeConfiguration(code=\"src\", scoring_script=\"score.py\")\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Let's start the package operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "build_package"
   },
   "outputs": [],
   "source": [
    "model_package = ml_client.models.package(model_name, model.version, pakage_config)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "The result of the package operation is an environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1682094844089
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "model_package"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 4.3 Deploy the package"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Now, we can deploy this package in an Online Endpoint. In this case, we will create a different deployment under the same endpoint so you can compare the two of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "configure_deployment"
   },
   "outputs": [],
   "source": [
    "deployment_package = ManagedOnlineDeployment(\n",
    "    name=\"with-package\",\n",
    "    endpoint_name=endpoint_name,\n",
    "    environment=model_package,\n",
    "    instance_count=1,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "> Tip: Notice how model or scoring script are not being indicated in this example. All of them are part of the package."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Let's create the deployment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create_deployment"
   },
   "outputs": [],
   "source": [
    "ml_client.online_deployments.begin_create_or_update(deployment_package).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 4.4 Test the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "test_deployment"
   },
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=endpoint_name,\n",
    "    deployment_name=\"with-package\",\n",
    "    request_file=\"sample-request.json\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "The method should return the prediction for two instances of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Deploying with packages directly from the deployment\n",
    "\n",
    "If you don't need to configure how the package is performed, you can quickly take advantage of the packaging functionality by indicating Online Endpoints to package before performing the deployment.\n",
    "\n",
    "To do so, indicate the argument `with_package=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "configure_deployment_with_package"
   },
   "outputs": [],
   "source": [
    "deployment_package = ManagedOnlineDeployment(\n",
    "    name=\"with-package-inline\",\n",
    "    endpoint_name=endpoint_name,\n",
    "    environment=base_environment,\n",
    "    model=model.id,\n",
    "    code_configuration=CodeConfiguration(\n",
    "            code='src', \n",
    "            scoring_script='score.py'\n",
    "        )\n",
    "    instance_count=1,\n",
    "    with_package=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create the deployment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create_deployment_with_package"
   },
   "outputs": [],
   "source": [
    "ml_client.online_deployments.begin_create_or_update(deployment_package).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## 6. Deploy the model outside AzureML\n",
    "\n",
    "Packages also allow you to generate a serving image that can be used to get the model deployed to whatever target is needed. Since AzureML is able to do some optimizations when packaging models, like for instance to mount the location where the model is stored rather than to copy the entire file inside of the image, you need to indicate to AzureML that you intend to take the image outside of the workspace.\n",
    "\n",
    "Let's see how you can configure the package to do so:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 5.1 Package the model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "We are going to package the model again, but this time, we are going to copy the resources inside of the image so we can take it outside of AzureML."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "configure_deployment_copy"
   },
   "outputs": [],
   "source": [
    "pakage_config = ModelPackage(\n",
    "    target_environment=\"sklearn-regression-online-pkg\",\n",
    "    base_environment_source=BaseEnvironment(\n",
    "        type=\"asset\",\n",
    "        resource_id=f\"azureml:{base_environment.name}:{base_environment.version}\",\n",
    "    ),\n",
    "    inferencing_server=AzureMLOnlineInferencingServer(\n",
    "        code_configuration=CodeConfiguration(code=\"src\", scoring_script=\"score.py\")\n",
    "    ),\n",
    "    model_configuration=ModelConfiguration(mode=\"copy\"),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Let's start the package operation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create_deployment_copy"
   },
   "outputs": [],
   "source": [
    "model_package = ml_client.models.package(model_name, model.version, pakage_config)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Clean un resources\n",
    "\n",
    "Once done, delete the associated resources from the workspace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "delete_resources"
   },
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.begin_delete(endpoint.name).result()"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "previews"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "microsoft": {
   "ms_spell_check": {
    "ms_spell_check_language": "en"
   }
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "7bc894e00fe86b2e284ef20cdeac20aaafcfa957033ee2bd5bf4c264a294a7ee"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}