bqml/06_model_deployment.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ur8xi4C7S06n"
   },
   "outputs": [],
   "source": [
    "# Copyright 2023 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAPoU8Sm5E6e"
   },
   "source": [
    "# FraudFinder - BigQuery ML - Model Deployment\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/ai-platform/notebooks/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/fraudfinder/raw/main/bqml/06_model_monitoring.ipynb\">\n",
    "       <img src=\"https://www.gstatic.com/cloud/images/navigation/vertex-ai.svg\" alt=\"Google Cloud Notebooks\">Open in Cloud Notebook\n",
    "    </a>\n",
    "  </td> \n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/fraudfinder/blob/main/bqml/06_model_monitoring.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Open in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/fraudfinder/blob/main/bqml/06_model_monitoring.ipynb\">\n",
    "        <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO"
   },
   "source": [
    "## Overview\n",
    "\n",
    "[FraudFinder](https://github.com/googlecloudplatform/fraudfinder) is a series of labs on how to build a real-time fraud detection system on Google Cloud. Throughout the FraudFinder labs, you will learn how to read historical bank transaction data stored in data warehouse, read from a live stream of new transactions, perform exploratory data analysis (EDA), do feature engineering, ingest features into a feature store, train a model using feature store, register your model in a model registry, evaluate your model, deploy your model to an endpoint, do real-time inference on your model with feature store, and monitor your model.\n",
    "\n",
    "### Objective\n",
    "\n",
    "In this notebook, you learn to deploy ML models and enable Vertex AI Model Monitoring service to detect feature skew and drift in the input predict requests. You will leverage the automatic generation of the input schema provided by Vertex AI Model Monitoring to analyze request data and detect feature skew and drift.\n",
    "\n",
    "\n",
    "This tutorial uses the following Google Cloud services:\n",
    "\n",
    "- [BigQuery](https://cloud.google.com/bigquery/)\n",
    "- [Vertex AI](https://cloud.google.com/vertex-ai/)\n",
    "\n",
    "The steps performed include:\n",
    "\n",
    "- Configure Vertex Explainable AI for feature attribution skew and drift detection.\n",
    "- Deploy the Vertex AI Model to a Vertex AI Endpoint.\n",
    "- Define and create a Model Monitoring job.\n",
    "\n",
    "### Costs \n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "\n",
    "* BigQuery\n",
    "* Vertex AI\n",
    "\n",
    "Learn about [BigQuery Pricing](https://cloud.google.com/bigquery/pricing), [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ze4-nDLfK4pw"
   },
   "source": [
    "### Load configuration settings from the setup notebook\n",
    "\n",
    "Set the constants used in this notebook and load the config settings from the `00_environment_setup.ipynb` notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gCuSR8GkAgzl"
   },
   "outputs": [],
   "source": [
    "GCP_PROJECTS = !gcloud config get-value project\n",
    "PROJECT_ID = GCP_PROJECTS[0]\n",
    "BUCKET_NAME = f\"{PROJECT_ID}-fraudfinder\"\n",
    "config = !gsutil cat gs://{BUCKET_NAME}/config/notebook_env.py\n",
    "print(config.n)\n",
    "exec(config.n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [],
   "source": [
    "# General\n",
    "from typing import Union, List, Dict\n",
    "from datetime import datetime, timedelta\n",
    "import time\n",
    "import random\n",
    "import pandas as pd\n",
    "\n",
    "# BigQuery\n",
    "from google.cloud import bigquery\n",
    "\n",
    "# Vertex AI \n",
    "from google.cloud import aiplatform as vertex_ai\n",
    "from google.cloud.aiplatform import model_monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BQ_DATASET = \"tx\"\n",
    "END_DATE_TRAIN = (datetime.today() - timedelta(days=1)).strftime(\"%Y-%m-%d\")\n",
    "TRAIN_TABLE_NAME = f\"train_table_{END_DATE_TRAIN.replace('-', '')}\"\n",
    "BQML_MODEL_ID = f\"{PROJECT_ID}.{BQ_DATASET}.{MODEL_NAME}\"\n",
    "MODEL_ARTIFACT_URI = f\"gs://{BUCKET_NAME}/deliverables/{MODEL_NAME}\"\n",
    "DEPLOY_VERSION = \"tf2-cpu.2-5\"\n",
    "DEPLOY_IMAGE = \"{}-docker.pkg.dev/vertex-ai/prediction/{}:latest\".format(\n",
    "    REGION.split(\"-\")[0], DEPLOY_VERSION\n",
    ")\n",
    "DEPLOY_MACHINE_TYPE = \"n1-standard-4\"\n",
    "MIN_REPLICA_COUNT = 1\n",
    "MAX_REPLICA_COUNT = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "init_aip:mbsdk,all"
   },
   "source": [
    "### Initialize Vertex AI and BigQuery SDKs for Python\n",
    "\n",
    "Initialize the Vertex AI SDK for Python for your project and corresponding bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "init_aip:mbsdk,all"
   },
   "outputs": [],
   "source": [
    "vertex_ai.init(project=PROJECT_ID, location=REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "83859376c893"
   },
   "source": [
    "Create the BigQuery client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0ab485806b17"
   },
   "outputs": [],
   "source": [
    "bq_client = bigquery.Client(project=PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor your model with Vertex AI Model Monitoring\n",
    "\n",
    "With Vertex AI Model Monitoring, you can monitor for skew and drift detection of the predictions, features and its attributions (Explainable AI) in the incoming prediction requests.\n",
    "\n",
    "With custom models, the model monitoring service requires:\n",
    "\n",
    "- for drift detection, the schema of the features to derive the feature values\n",
    "\n",
    "- for skew detection, a training data sample as baseline to calculate the distribution\n",
    "\n",
    "- for feature attribution skew and drift detection, Vertex Explainable AI to be configured. \n",
    "\n",
    "In the following sections, we are going to cover all those requirements settings. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Configure Vertex Explainable AI for feature attribution skew and drift detection\n",
    "\n",
    "To configure Vertex Explainable AI for feature attribution skew and drift detection in our case, you need to define the explainability specification. Then, you need to pass the explainability specification to the model monitoring job. \n",
    "\n",
    "To define the explainability specification, you get the ML model schema from BQML model. Next you configure parameters for explaining model's predictions and metadata for describing expected model input and output. Please check out the [Vertex Explainable AI documentation](https://cloud.google.com/vertex-ai/docs/explainable-ai/configuring-explanations-feature-based#upload_model_xai_tf_sampled_shapley-gcloud) to know more about it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bqml_model = bq_client.get_model(BQML_MODEL_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "index_feature_mapping = []\n",
    "for feature in bqml_model.feature_columns:\n",
    "    index_feature_mapping.append(feature.name)\n",
    "label_name = bqml_model.label_columns[0].name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explanation_params =  vertex_ai.explain.ExplanationParameters({\"sampled_shapley_attribution\": {\"path_count\": 10}})\n",
    "explanation_inputs = {feature_name:{'input_tensor_name':feature_name} for feature_name in index_feature_mapping}\n",
    "explanation_outputs = {label_name: {'output_tensor_name': label_name}}\n",
    "explanation_metadata = vertex_ai.explain.ExplanationMetadata(inputs=explanation_inputs, outputs=explanation_outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploy the Vertex AI Model to a Vertex AI Endpoint\n",
    "\n",
    "After you define the explainability specification, you can deploy the model previously trained using Vertex AI ML pipeline to a Vertex AI Endpoint. You get the model resource from the Vertex AI Model Registry. Then you deploy the model with the explainability specification in a new endpoint. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = vertex_ai.Model.list(filter=f\"display_name=bqml_fraud_classifier\")[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint = vertex_ai.Endpoint.create(display_name = f\"{ENDPOINT_NAME}_monitored\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.deploy(\n",
    "        endpoint=endpoint,\n",
    "        deployed_model_display_name=\"fraud_detector_\" + ID,\n",
    "        machine_type=DEPLOY_MACHINE_TYPE,\n",
    "        min_replica_count=MIN_REPLICA_COUNT,\n",
    "        max_replica_count=MAX_REPLICA_COUNT,\n",
    "        explanation_parameters=explanation_params,\n",
    "        explanation_metadata=explanation_metadata,\n",
    "        traffic_percentage = 100,\n",
    "        sync=True\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define and create a Model Monitoring job\n",
    "\n",
    "To set up either skew detection or drift detection, create a model deployment monitoring job. \n",
    "\n",
    "The job requires the following specifications:\n",
    "\n",
    "- `alert_config`: Configures how alerts are sent to the user. Right now only email alert is supported.\n",
    "- `schedule_config`: Configures model monitoring job scheduling interval in hours. This defines how often the monitoring jobs are triggered.\n",
    "- `logging_sampling_strategy`: Sample Strategy for logging.\n",
    "- `drift_config` : Configures drift thresholds per each feature to monitor.\n",
    "- `skew_config` : Configures skew thresholds per each feature to monitor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Define the alerting configuration\n",
    "\n",
    "The alerting configuration contains the mails to send alerts to. Also you can use the configuration to stream anomalies to Cloud Logging. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "USER_EMAILS = ['recipient1@domain.com'] #'recipient1@domain.com', 'recipient2@domain.com'\n",
    "alert_config = model_monitoring.EmailAlertConfig(USER_EMAILS, enable_logging=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Define the schedule configuration\n",
    "\n",
    "The schedule configuration sets the hourly model monitoring job scheduling interval. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MONITOR_INTERVAL = 1\n",
    "schedule_config = model_monitoring.ScheduleConfig(monitor_interval=MONITOR_INTERVAL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Define the logging sample strategy\n",
    "\n",
    "With the logging sample strategy, you configure how the model monitoring service randomly sample predictions to calculate monitoring metrics. The selected samples are logged to a BigQuery table. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SAMPLE_RATE = 0.5 \n",
    "logging_sampling_strategy = model_monitoring.RandomSampleConfig(sample_rate=SAMPLE_RATE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Define the drift detection configuration\n",
    "\n",
    "With the drift detection configuration, you define the input features and the associated thresholds for monitoring feature distribution drift and feature attribution drift. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DRIFT_THRESHOLD_VALUE = 0.05\n",
    "ATTRIBUTION_DRIFT_THRESHOLD_VALUE = 0.05\n",
    "\n",
    "drift_thresholds = {\n",
    "    \"tx_amount\": DRIFT_THRESHOLD_VALUE,\n",
    "    \"customer_id_nb_tx_1day_window\": DRIFT_THRESHOLD_VALUE,\n",
    "    \"customer_id_avg_amount_1day_window\": DRIFT_THRESHOLD_VALUE,\n",
    "    \"customer_id_nb_tx_15min_window\": DRIFT_THRESHOLD_VALUE,\n",
    "    \"customer_id_avg_amount_15min_window\": DRIFT_THRESHOLD_VALUE,\n",
    "    \"terminal_id_nb_tx_1day_window\": DRIFT_THRESHOLD_VALUE,\n",
    "    \"terminal_id_risk_1day_window\": DRIFT_THRESHOLD_VALUE,\n",
    "    \"terminal_id_nb_tx_15min_window\": DRIFT_THRESHOLD_VALUE,\n",
    "    \"terminal_id_avg_amount_15min_window\": DRIFT_THRESHOLD_VALUE\n",
    "}\n",
    "\n",
    "attribution_drift_thresholds = {\n",
    "    \"tx_amount\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE,\n",
    "    \"customer_id_nb_tx_1day_window\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE,\n",
    "    \"customer_id_avg_amount_1day_window\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE,\n",
    "    \"customer_id_nb_tx_15min_window\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE,\n",
    "    \"customer_id_avg_amount_15min_window\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE,\n",
    "    \"terminal_id_nb_tx_1day_window\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE,\n",
    "    \"terminal_id_risk_1day_window\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE,\n",
    "    \"terminal_id_nb_tx_15min_window\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE,\n",
    "    \"terminal_id_avg_amount_15min_window\": ATTRIBUTION_DRIFT_THRESHOLD_VALUE\n",
    "}\n",
    "\n",
    "drift_config = model_monitoring.DriftDetectionConfig(\n",
    "    drift_thresholds=drift_thresholds,\n",
    "    attribute_drift_thresholds=attribution_drift_thresholds,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Define the skew detection configuration\n",
    "\n",
    "With the skew detection configuration, you define the input features and the associated thresholds for monitoring feature distribution skew and feature attribution skew."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_DATA_SOURCE_URI = f\"bq://{PROJECT_ID}.{BQ_DATASET}.{TRAIN_TABLE_NAME}\"\n",
    "TARGET = \"tx_fraud\"\n",
    "SKEW_THRESHOLD_VALUE = 0.5\n",
    "ATTRIBUTE_SKEW_THRESHOLD_VALUE = 0.5\n",
    "\n",
    "skew_thresholds = {\n",
    "    \"tx_amount\": SKEW_THRESHOLD_VALUE,\n",
    "    \"customer_id_nb_tx_1day_window\": SKEW_THRESHOLD_VALUE,\n",
    "    \"customer_id_avg_amount_1day_window\": SKEW_THRESHOLD_VALUE,\n",
    "    \"customer_id_nb_tx_15min_window\": SKEW_THRESHOLD_VALUE,\n",
    "    \"customer_id_avg_amount_15min_window\": SKEW_THRESHOLD_VALUE,\n",
    "    \"terminal_id_nb_tx_1day_window\": SKEW_THRESHOLD_VALUE,\n",
    "    \"terminal_id_risk_1day_window\": SKEW_THRESHOLD_VALUE,\n",
    "    \"terminal_id_nb_tx_15min_window\": SKEW_THRESHOLD_VALUE,\n",
    "    \"terminal_id_avg_amount_15min_window\": SKEW_THRESHOLD_VALUE\n",
    "}\n",
    "\n",
    "attribute_skew_thresholds = {\n",
    "    \"tx_amount\": ATTRIBUTE_SKEW_THRESHOLD_VALUE,\n",
    "    \"customer_id_nb_tx_1day_window\": ATTRIBUTE_SKEW_THRESHOLD_VALUE,\n",
    "    \"customer_id_avg_amount_1day_window\": ATTRIBUTE_SKEW_THRESHOLD_VALUE,\n",
    "    \"customer_id_nb_tx_15min_window\": ATTRIBUTE_SKEW_THRESHOLD_VALUE,\n",
    "    \"customer_id_avg_amount_15min_window\": ATTRIBUTE_SKEW_THRESHOLD_VALUE,\n",
    "    \"terminal_id_nb_tx_1day_window\": ATTRIBUTE_SKEW_THRESHOLD_VALUE,\n",
    "    \"terminal_id_risk_1day_window\": ATTRIBUTE_SKEW_THRESHOLD_VALUE,\n",
    "    \"terminal_id_nb_tx_15min_window\": ATTRIBUTE_SKEW_THRESHOLD_VALUE,\n",
    "    \"terminal_id_avg_amount_15min_window\": ATTRIBUTE_SKEW_THRESHOLD_VALUE\n",
    "}\n",
    "\n",
    "skew_config = model_monitoring.SkewDetectionConfig(\n",
    "    data_source=TRAIN_DATA_SOURCE_URI,\n",
    "    skew_thresholds=skew_thresholds,\n",
    "    attribute_skew_thresholds=attribute_skew_thresholds,\n",
    "    target_field=TARGET,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create the job configuration\n",
    "\n",
    "Last step you create the Model monitoring job configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explanation_config = model_monitoring.ExplanationConfig()\n",
    "\n",
    "objective_config = model_monitoring.ObjectiveConfig(\n",
    "    skew_detection_config=skew_config,\n",
    "    drift_detection_config=drift_config,\n",
    "    explanation_config=explanation_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create the model monitoring job\n",
    "\n",
    "Now you can create the model monitoring job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "monitoring_job = vertex_ai.ModelDeploymentMonitoringJob.create(\n",
    "    display_name=\"fraud_detection_\" + ID,\n",
    "    project=PROJECT_ID,\n",
    "    location=REGION,\n",
    "    endpoint=endpoint,\n",
    "    logging_sampling_strategy=logging_sampling_strategy,\n",
    "    schedule_config=schedule_config,\n",
    "    alert_config=alert_config,\n",
    "    objective_configs=objective_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check the monitoring job state\n",
    "\n",
    "You can check the status of the model monitoring job using the state attribute of the job instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jobs = monitoring_job.list(filter=f\"display_name=fraud_detection_{ID}\")\n",
    "job = jobs[0]\n",
    "print(job.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Receiving email alert\n",
    "\n",
    "After a minute or two, you should receive email at the address you configured above for `USER_EMAIL`. This email confirms successful deployment of your monitoring job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Monitoring results in the Cloud Console\n",
    "\n",
    "After one hour, you can examine your model monitoring data from the Cloud Console."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### END\n",
    "\n",
    "Congrats! You successully finished the Fraudfinder lab series on how to build a real-time fraud detection system on Google Cloud. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (DO NOT RUN) Cleaning up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Delete the monitoring job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# monitoring_job.pause()\n",
    "# monitoring_job.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Undeploy the model and delete the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# endpoint.undeploy_all()\n",
    "# endpoint.delete()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "bqml-online-prediction.ipynb",
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "common-cpu.m104",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/base-cpu:m104"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}