retail/recommendation-system/bqml-scann/ann01_create_index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Low-latency item-to-item recommendation system - Creating ANN index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This notebook is a part of the series that describes the process of implementing a [**Low-latency item-to-item recommendation system**](https://github.com/jarokaz/analytics-componentized-patterns/tree/master/retail/recommendation-system/bqml-scann).\n",
    "\n",
    "The notebook demonstrates how to create and deploy an ANN index using item embeddings created in the preceding notebooks. In this notebook you go through the following steps.\n",
    "\n",
    "1. Exporting embeddings from BigQuery into the JSONL formated file.\n",
    "2. Creating an ANN Index using the exported embeddings.\n",
    "3. Creating and ANN Endpoint. \n",
    "4. Deploying the ANN Index to the ANN Endpoint.\n",
    "5. Testing the deployed ANN Index.\n",
    "\n",
    "This notebook was designed to run on [AI Platform Notebooks](https://cloud.google.com/ai-platform-notebooks). Before running the notebook make sure that you have completed the setup steps as described in the [README file](README.md).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the notebook's environment\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import notebook dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import datetime\n",
    "import logging\n",
    "import os\n",
    "import json\n",
    "import pandas as pd\n",
    "import time\n",
    "import sys\n",
    "\n",
    "import grpc\n",
    "\n",
    "import google.auth\n",
    "import numpy as np\n",
    "import tensorflow.io as tf_io\n",
    "\n",
    "from google.cloud import bigquery\n",
    "from typing import List, Optional, Text, Tuple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the experimental release, the *Online Querying API* of the ANN service is exposed throught the GRPC interface. The `ann_grpc` folder contains the grpc client stub to interface to the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANN_GRPC_ENDPOINT_STUB = 'ann_grpc'\n",
    "if ANN_GRPC_ENDPOINT_STUB not in sys.path:\n",
    "    sys.path.append(ANN_GRPC_ENDPOINT_STUB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ann_grpc.match_pb2_grpc as match_pb2_grpc\n",
    "import ann_grpc.match_pb2 as match_pb2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure GCP environment\n",
    "\n",
    "Set the following constants to the values reflecting your environment:\n",
    "\n",
    "* `PROJECT_ID` - your GCP project ID.\n",
    "* `PROJECT_NUMBER` - your GCP project number.\n",
    "* `BQ_DATASET_NAME` - the name of the BigQuery dataset that contains the item embeddings table.\n",
    "* `BQ_LOCATION` - the dataset location\n",
    "* `DATA_LOCATION` - a GCS location for the exported embeddings (JSONL) files.\n",
    "* `VPC_NAME` - a name of the GCP VPC to use for the index deployments. Use the name of the VPC prepared during the initial setup. \n",
    "* `REGION` - a compute region. Don't change the default - `us-central` - while the ANN Service is in the experimental stage\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = 'jk-mlops-dev' # <-CHANGE THIS\n",
    "PROJECT_NUMBER = '895222332033' # <-CHANGE THIS\n",
    "BQ_DATASET_NAME = 'song_embeddings' # <- CHANGE THIS\n",
    "BQ_LOCATION = 'US' # <- CHANGE THIS\n",
    "DATA_LOCATION = 'gs://jk-ann-staging/embeddings' # <-CHANGE THIS\n",
    "VPC_NAME = 'default' # <-CHANGE THIS\n",
    "\n",
    "EMBEDDINGS_TABLE = 'item_embeddings'\n",
    "REGION = 'us-central1'\n",
    "MATCH_SERVICE_PORT = 10000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting the embeddings\n",
    "\n",
    "In the preceeding notebooks you trained the Matrix Factorization BQML model and exported the embeddings to the `item_embeddings` table. \n",
    "\n",
    "In this step you will extract the embeddings to a set of JSONL files in the format required by the ANN service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the number of embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = bigquery.Client(project=PROJECT_ID, location=BQ_LOCATION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>embedding_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>35519</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   embedding_count\n",
       "0            35519"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = f\"\"\"\n",
    "    SELECT COUNT(*) embedding_count\n",
    "    FROM {BQ_DATASET_NAME}.item_embeddings;\n",
    "\"\"\"\n",
    "\n",
    "query_job = client.query(query)\n",
    "query_job.to_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Export the embeddings\n",
    "\n",
    "You will use the [BigQuery export job](https://cloud.google.com/bigquery/docs/exporting-data) to export the embeddings table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<google.cloud.bigquery.job.ExtractJob at 0x7f2e4841f0d0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_name_pattern = 'embedding-*.json'\n",
    "destination_uri = f'{DATA_LOCATION}/{file_name_pattern}'\n",
    "table_id = 'item_embeddings'\n",
    "destination_format = 'NEWLINE_DELIMITED_JSON'\n",
    "\n",
    "dataset_ref = bigquery.DatasetReference(PROJECT_ID, BQ_DATASET_NAME)\n",
    "table_ref = dataset_ref.table(table_id)\n",
    "job_config = bigquery.job.ExtractJobConfig()\n",
    "job_config.destination_format = bigquery.DestinationFormat.NEWLINE_DELIMITED_JSON\n",
    "\n",
    "extract_job = client.extract_table(\n",
    "    table_ref,\n",
    "    destination_uris=destination_uri,\n",
    "    job_config=job_config,\n",
    "    #location=BQ_LOCATION,\n",
    ")  \n",
    "\n",
    "extract_job.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the extracted files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://jk-ann-staging/embeddings/embedding-000000000000.json\n"
     ]
    }
   ],
   "source": [
    "! gsutil ls {DATA_LOCATION}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an ANN index deployment\n",
    "\n",
    "Deploying an ANN index is a 3 step process:\n",
    "1. Creating an index from source files\n",
    "2. Creating an endpoint to access the index\n",
    "3. Deploying the index to the endpoint\n",
    "\n",
    "\n",
    "You will use the REST interface to invoke the AI Platform ANN Service Control Plane API that manages indexes, endpoints, and deployments.\n",
    "\n",
    "After the index has been deployed you can submit matching requests using Online Querying API. In the experimental stage this API is only accessible through the gRPC interface.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define helper classes to encapsulate the ANN Service REST API.\n",
    "\n",
    "Currently, there is no Python client that encapsulates the ANN Service Control Plane API. The below code snippet defines a simple wrapper that encapsulates a subset of REST APIs used in this notebook.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import logging\n",
    "import json\n",
    "import time\n",
    "\n",
    "import google.auth\n",
    "\n",
    "class ANNClient(object):\n",
    "    \"\"\"Base ANN Service client.\"\"\"\n",
    "    \n",
    "    def __init__(self, project_id, project_number, region):\n",
    "        credentials, _ = google.auth.default()\n",
    "        self.authed_session = google.auth.transport.requests.AuthorizedSession(credentials)\n",
    "        self.ann_endpoint = f'{region}-aiplatform.googleapis.com'\n",
    "        self.ann_parent = f'https://{self.ann_endpoint}/v1alpha1/projects/{project_id}/locations/{region}'\n",
    "        self.project_id = project_id\n",
    "        self.project_number = project_number\n",
    "        self.region = region\n",
    "        \n",
    "    def wait_for_completion(self, operation_id, message, sleep_time):\n",
    "        \"\"\"Waits for a completion of a long running operation.\"\"\"\n",
    "        \n",
    "        api_url = f'{self.ann_parent}/operations/{operation_id}'\n",
    "\n",
    "        start_time = datetime.datetime.utcnow()\n",
    "        while True:\n",
    "            response = self.authed_session.get(api_url)\n",
    "            if response.status_code != 200:\n",
    "                raise RuntimeError(response.json())\n",
    "            if 'done' in response.json().keys():\n",
    "                logging.info('Operation completed!')\n",
    "                break\n",
    "            elapsed_time = datetime.datetime.utcnow() - start_time\n",
    "            logging.info('{}. Elapsed time since start: {}.'.format(\n",
    "                message, str(elapsed_time)))\n",
    "            time.sleep(sleep_time)\n",
    "    \n",
    "        return response.json()['response']\n",
    "\n",
    "\n",
    "class IndexClient(ANNClient):\n",
    "    \"\"\"Encapsulates a subset of control plane APIs \n",
    "    that manage ANN indexes.\"\"\"\n",
    "\n",
    "    def __init__(self, project_id, project_number, region):\n",
    "        super().__init__(project_id, project_number, region)\n",
    "\n",
    "    def create_index(self, display_name, description, metadata):\n",
    "        \"\"\"Creates an ANN Index.\"\"\"\n",
    "    \n",
    "        api_url = f'{self.ann_parent}/indexes'\n",
    "    \n",
    "        request_body = {\n",
    "            'display_name': display_name,\n",
    "            'description': description,\n",
    "            'metadata': metadata\n",
    "        }\n",
    "    \n",
    "        response = self.authed_session.post(api_url, data=json.dumps(request_body))\n",
    "        if response.status_code != 200:\n",
    "            raise RuntimeError(response.text)\n",
    "        operation_id = response.json()['name'].split('/')[-1]\n",
    "        \n",
    "        return operation_id\n",
    "\n",
    "    def list_indexes(self, display_name=None):\n",
    "        \"\"\"Lists all indexes with a given display name or\n",
    "        all indexes if the display_name is not provided.\"\"\"\n",
    "    \n",
    "        if display_name:\n",
    "            api_url = f'{self.ann_parent}/indexes?filter=display_name=\"{display_name}\"'\n",
    "        else:\n",
    "            api_url = f'{self.ann_parent}/indexes'\n",
    "\n",
    "        response = self.authed_session.get(api_url).json()\n",
    "\n",
    "        return response['indexes'] if response else []\n",
    "    \n",
    "    def delete_index(self, index_id):\n",
    "        \"\"\"Deletes an ANN index.\"\"\"\n",
    "        \n",
    "        api_url = f'{self.ann_parent}/indexes/{index_id}'\n",
    "        response = self.authed_session.delete(api_url)\n",
    "        if response.status_code != 200:\n",
    "            raise RuntimeError(response.text)\n",
    "\n",
    "\n",
    "class IndexDeploymentClient(ANNClient):\n",
    "    \"\"\"Encapsulates a subset of control plane APIs \n",
    "    that manage ANN endpoints and deployments.\"\"\"\n",
    "    \n",
    "    def __init__(self, project_id, project_number, region):\n",
    "        super().__init__(project_id, project_number, region)\n",
    "\n",
    "    def create_endpoint(self, display_name, vpc_name):\n",
    "        \"\"\"Creates an ANN endpoint.\"\"\"\n",
    "    \n",
    "        api_url = f'{self.ann_parent}/indexEndpoints'\n",
    "        network_name = f'projects/{self.project_number}/global/networks/{vpc_name}'\n",
    "\n",
    "        request_body = {\n",
    "            'display_name': display_name,\n",
    "            'network': network_name\n",
    "        }\n",
    "\n",
    "        response = self.authed_session.post(api_url, data=json.dumps(request_body))\n",
    "        if response.status_code != 200:\n",
    "            raise RuntimeError(response.text)\n",
    "        operation_id = response.json()['name'].split('/')[-1]\n",
    "    \n",
    "        return operation_id\n",
    "    \n",
    "    def list_endpoints(self, display_name=None):\n",
    "        \"\"\"Lists all ANN endpoints with a given display name or\n",
    "        all endpoints in the project if the display_name is not provided.\"\"\"\n",
    "        \n",
    "        if display_name:\n",
    "            api_url = f'{self.ann_parent}/indexEndpoints?filter=display_name=\"{display_name}\"'\n",
    "        else:\n",
    "            api_url = f'{self.ann_parent}/indexEndpoints'\n",
    "\n",
    "        response = self.authed_session.get(api_url).json()\n",
    " \n",
    "        return response['indexEndpoints'] if response else []\n",
    "    \n",
    "    def delete_endpoint(self, endpoint_id):\n",
    "        \"\"\"Deletes an ANN endpoint.\"\"\"\n",
    "        \n",
    "        api_url = f'{self.ann_parent}/indexEndpoints/{endpoint_id}'\n",
    "        \n",
    "        response = self.authed_session.delete(api_url)\n",
    "        if response.status_code != 200:\n",
    "            raise RuntimeError(response.text)\n",
    "        \n",
    "        return response.json()\n",
    "    \n",
    "    def create_deployment(self, display_name, deployment_id, endpoint_id, index_id):\n",
    "        \"\"\"Deploys an ANN index to an endpoint.\"\"\"\n",
    "    \n",
    "        api_url = f'{self.ann_parent}/indexEndpoints/{endpoint_id}:deployIndex'\n",
    "        index_name = f'projects/{self.project_number}/locations/{self.region}/indexes/{index_id}'\n",
    "\n",
    "        request_body = {\n",
    "            'deployed_index': {\n",
    "                'id': deployment_id,\n",
    "                'index': index_name,\n",
    "                'display_name': display_name\n",
    "            }\n",
    "        }\n",
    "\n",
    "        response = self.authed_session.post(api_url, data=json.dumps(request_body))\n",
    "        if response.status_code != 200:\n",
    "            raise RuntimeError(response.text)\n",
    "        operation_id = response.json()['name'].split('/')[-1]\n",
    "        \n",
    "        return operation_id\n",
    "    \n",
    "    def get_deployment_grpc_ip(self, endpoint_id, deployment_id):\n",
    "        \"\"\"Returns a private IP address for a gRPC interface to \n",
    "        an Index deployment.\"\"\"\n",
    "  \n",
    "        api_url = f'{self.ann_parent}/indexEndpoints/{endpoint_id}'\n",
    "\n",
    "        response = self.authed_session.get(api_url)\n",
    "        if response.status_code != 200:\n",
    "            raise RuntimeError(response.text)\n",
    "            \n",
    "        endpoint_ip = None\n",
    "        if 'deployedIndexes' in response.json().keys():\n",
    "            for deployment in response.json()['deployedIndexes']:\n",
    "                if deployment['id'] == deployment_id:\n",
    "                    endpoint_ip = deployment['privateEndpoints']['matchGrpcAddress']\n",
    "                    \n",
    "        return endpoint_ip\n",
    "\n",
    "    \n",
    "    def delete_deployment(self, endpoint_id, deployment_id):\n",
    "        \"\"\"Undeployes an index from an endpoint.\"\"\"\n",
    "        \n",
    "        api_url = f'{self.ann_parent}/indexEndpoints/{endpoint_id}:undeployIndex'\n",
    "        \n",
    "        request_body = {\n",
    "            'deployed_index_id': deployment_id\n",
    "        }\n",
    "    \n",
    "        response = self.authed_session.post(api_url, data=json.dumps(request_body))\n",
    "        if response.status_code != 200:\n",
    "            raise RuntimeError(response.text)\n",
    "        \n",
    "        return response\n",
    "    \n",
    "index_client = IndexClient(PROJECT_ID, PROJECT_NUMBER, REGION)\n",
    "deployment_client = IndexDeploymentClient(PROJECT_ID, PROJECT_NUMBER, REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an ANN index\n",
    "\n",
    "#### List all indexes registered with the ANN service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are not any indexes registered with the service\n"
     ]
    }
   ],
   "source": [
    "indexes = index_client.list_indexes()\n",
    "\n",
    "if not indexes:\n",
    "    print('There are not any indexes registered with the service')\n",
    "\n",
    "for index in indexes:\n",
    "    print(index['name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Configure and create a new index based on the exported embeddings\n",
    "\n",
    "Index creation is a long running operation. Be patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "index_display_name = 'Song embeddings'\n",
    "index_description = 'Song embeddings created BQML Matrix Factorization model'\n",
    "\n",
    "index_metadata = {\n",
    "    'contents_delta_uri': DATA_LOCATION,\n",
    "    'config': {\n",
    "        'dimensions': 50,\n",
    "        'approximate_neighbors_count': 50,\n",
    "        'distance_measure_type': 'DOT_PRODUCT_DISTANCE',\n",
    "        'feature_norm_type': 'UNIT_L2_NORM',\n",
    "        'tree_ah_config': {\n",
    "            'child_node_count': 1000,\n",
    "            'max_leaves_to_search': 100\n",
    "         }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Creating index. Elapsed time since start: 0:00:00.284307.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:00:20.590597.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:00:40.895456.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:01:01.212661.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:01:21.541146.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:01:41.850597.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:02:02.007845.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:02:22.286154.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:02:42.594481.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:03:02.768118.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:03:23.046792.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:03:43.167440.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:04:03.450286.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:04:23.576374.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:04:43.861362.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:05:04.153150.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:05:24.271262.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:05:44.388816.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:06:04.500550.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:06:24.816035.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:06:44.921801.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:07:05.245659.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:07:25.388260.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:07:45.692861.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:08:05.998090.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:08:26.113222.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:08:46.253836.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:09:06.423070.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:09:26.701742.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:09:46.817432.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:10:07.115610.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:10:27.229846.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:10:47.518691.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:11:07.814059.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:11:28.202194.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:11:48.485845.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:12:08.603703.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:12:28.930780.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:12:49.167657.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:13:09.455059.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:13:29.596488.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:13:49.921589.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:14:10.489026.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:14:30.787925.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:14:51.157541.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:15:11.284806.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:15:31.405137.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:15:51.756367.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:16:12.004751.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:16:32.280723.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:16:52.573849.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:17:12.856906.\n",
      "INFO:root:Creating index. Elapsed time since start: 0:17:33.208577.\n",
      "INFO:root:Operation completed!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.Index', 'name': 'projects/895222332033/locations/us-central1/indexes/4272227196514336768'}\n"
     ]
    }
   ],
   "source": [
    "logging.getLogger().setLevel(logging.INFO)\n",
    "\n",
    "operation_id = index_client.create_index(index_display_name, \n",
    "                                          index_description,\n",
    "                                          index_metadata)\n",
    "\n",
    "response = index_client.wait_for_completion(operation_id, 'Creating index', 20)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verify that the index was created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/895222332033/locations/us-central1/indexes/4272227196514336768\n",
      "Index: 4272227196514336768 will be used for deployment\n"
     ]
    }
   ],
   "source": [
    "indexes = index_client.list_indexes(index_display_name)\n",
    "\n",
    "for index in indexes:\n",
    "    print(index['name'])\n",
    "\n",
    "if indexes: \n",
    "    index_id = index['name'].split('/')[-1]\n",
    "    print(f'Index: {index_id} will be used for deployment')\n",
    "else:\n",
    "    print('No indexes available for deployment')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the index deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List all endpoints registered with the ANN service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are not any endpoints registered with the service\n"
     ]
    }
   ],
   "source": [
    "endpoints = deployment_client.list_endpoints()\n",
    "\n",
    "if not endpoints:\n",
    "    print('There are not any endpoints registered with the service')\n",
    "\n",
    "for endpoint in endpoints:\n",
    "    print(endpoint['name'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create an index endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_display_name = 'Song embeddings endpoint'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for endpoint. Elapsed time since start: 0:00:00.275412.\n",
      "INFO:root:Waiting for endpoint. Elapsed time since start: 0:00:10.556182.\n",
      "INFO:root:Operation completed!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.IndexEndpoint', 'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160'}\n"
     ]
    }
   ],
   "source": [
    "operation_id = deployment_client.create_endpoint(deployment_display_name, VPC_NAME)\n",
    "\n",
    "response = index_client.wait_for_completion(operation_id, 'Waiting for endpoint', 10)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verify that the endpoint was created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160\n",
      "Endpoint: 4112349409742684160 will be used for deployment\n"
     ]
    }
   ],
   "source": [
    "endpoints = deployment_client.list_endpoints(deployment_display_name)\n",
    "\n",
    "for endpoint in endpoints:\n",
    "    print(endpoint['name'])\n",
    "    \n",
    "if endpoints: \n",
    "    endpoint_id = endpoint['name'].split('/')[-1]\n",
    "    print(f'Endpoint: {endpoint_id} will be used for deployment')\n",
    "else:\n",
    "    print('No endpoints available for deployment')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Deploy the index to the endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Set the deployed index ID\n",
    "\n",
    "The ID of the deployed index must be unique within your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_display_name = 'Song embeddings deployed index'\n",
    "deployed_index_id = 'songs_embeddings_deployed_index'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Deploy the index\n",
    "\n",
    "Be patient. Index deployment is a long running operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:00.300974.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:10.608720.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:20.916550.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:31.250512.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:41.539311.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:51.832651.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:02.144021.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:12.452319.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:22.717381.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:33.029002.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:43.170426.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:53.430804.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:03.600185.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:13.966494.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:24.259920.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:34.554471.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:44.677820.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:54.813466.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:05.096286.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:15.293527.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:25.569682.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:35.850643.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:46.129687.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:56.433004.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:06.553300.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:16.865053.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:27.131709.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:37.267542.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:47.375035.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:57.661588.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:07.958157.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:18.054951.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:28.319167.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:38.432195.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:48.722748.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:58.853269.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:09.131452.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:19.418657.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:29.533979.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:39.809766.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:50.081380.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:00.201212.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:10.497984.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:20.849624.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:31.141277.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:41.237883.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:51.503889.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:01.613211.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:11.727276.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:21.840908.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:31.968369.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:42.245167.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:52.519226.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:02.806466.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:13.124518.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:23.234916.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:33.361537.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:43.451962.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:53.553160.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:03.664825.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:13.795493.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:23.952488.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:34.080495.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:44.208095.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:54.353848.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:04.478361.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:14.621959.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:24.723540.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:34.836618.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:44.940823.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:55.067269.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:05.185345.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:15.324509.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:25.445491.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:35.536617.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:45.799675.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:55.927069.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:06.249747.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:16.369676.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:26.658924.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:36.765529.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'projects/895222332033/locations/us-central1/indexEndpoints/4112349409742684160/operations/9143743205747982336', 'metadata': {'@type': 'type.googleapis.com/google.cloud.aiplatform.v1alpha1.DeployIndexOperationMetadata', 'genericMetadata': {'createTime': '2021-01-14T20:19:24.354833Z', 'updateTime': '2021-01-14T20:19:24.354833Z'}}}\n"
     ]
    }
   ],
   "source": [
    "response = index_client.wait_for_completion(operation_id, 'Waiting for deployment', 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:00.309496.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:10.618832.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:20.953429.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:31.276869.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:41.582544.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:00:51.860203.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:02.145775.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:12.413776.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:22.769919.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:33.043710.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:43.331444.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:01:53.635462.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:03.913796.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:14.246248.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:24.438006.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:34.737109.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:45.061957.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:02:55.214420.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:05.503850.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:15.619386.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:25.763396.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:35.926869.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:46.206785.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:03:56.344440.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:06.661850.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:16.790771.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:27.100158.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:37.478707.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:47.790848.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:04:58.108535.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:08.379241.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:18.693843.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:28.969801.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:39.122504.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:49.430795.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:05:59.568054.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:09.933457.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:20.214257.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:30.500999.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:40.779993.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:06:51.158269.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:01.475487.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:11.603737.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:21.890999.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:32.167565.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:42.449151.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:07:52.745178.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:03.047037.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:13.190369.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:23.500509.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:33.640225.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:43.755012.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:08:54.051967.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:04.331984.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:14.495996.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:24.617013.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:34.892956.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:45.165349.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:09:55.271476.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:05.551907.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:15.696284.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:26.041164.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:36.310077.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:46.466342.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:10:56.574228.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:06.863240.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:16.965472.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:27.089647.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:37.181525.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:47.483513.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:11:57.588205.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:07.875761.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:18.003721.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:28.269875.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:38.527134.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:48.796227.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:12:59.091610.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:09.193477.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:19.279818.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:29.388218.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:39.487099.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:49.628959.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:13:59.883954.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:14:09.984956.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:14:20.108242.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:14:30.236006.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:14:40.592755.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:14:50.703748.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:15:00.847793.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:15:10.953756.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:15:21.107493.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:15:31.231147.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:15:41.334272.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:15:51.630401.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:16:01.733843.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:16:11.842532.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:16:21.949418.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:16:32.219992.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:16:42.518122.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:16:52.624872.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:17:02.896258.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:17:13.163637.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:17:23.462698.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:17:33.614831.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:17:43.734966.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:17:53.879712.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:18:03.987525.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:18:14.091399.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:18:24.196694.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:18:34.323345.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:18:44.498746.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:18:54.604675.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:19:04.730859.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:19:15.034432.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:19:25.186901.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:19:35.300179.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:19:45.578095.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:19:55.854867.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:20:06.188719.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:20:16.458125.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:20:26.759566.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:20:36.884720.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:20:47.229833.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:20:57.353245.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:21:07.469112.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:21:17.749214.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:21:28.010035.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:21:38.291132.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:21:58.556089.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:22:08.847796.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:22:18.973042.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:22:29.256783.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:22:39.523975.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:22:49.647099.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:22:59.779039.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:23:09.901494.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:23:20.024414.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:23:30.281691.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:23:40.411582.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:23:50.715438.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:24:00.825649.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:24:10.927193.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:24:21.204168.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:24:31.455315.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:24:41.641686.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:24:51.917218.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:25:02.204333.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:25:12.299840.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:25:22.597023.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:25:32.730363.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:25:42.868776.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:25:52.975587.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:26:03.290870.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:26:13.585602.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:26:23.726020.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:26:33.998132.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:26:44.283786.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:26:54.546448.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:27:04.661727.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:27:14.793746.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:27:25.076297.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:27:35.348142.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:27:45.497051.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:27:55.625915.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:28:05.748964.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:28:15.877515.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:28:26.164943.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:28:36.284533.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:28:46.414829.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:28:56.514943.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:29:06.662625.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:29:16.779856.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:29:26.917760.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:29:37.052899.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:29:47.182276.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:29:57.472446.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:30:07.592834.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:30:17.753531.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:30:27.850986.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:30:37.976432.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:30:48.098709.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:30:58.218721.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:31:08.348619.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:31:18.482850.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:31:28.594592.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:31:38.880292.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:31:49.007348.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:31:59.094694.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:32:09.304929.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:32:19.584198.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:32:29.706484.\n",
      "INFO:root:Waiting for deployment. Elapsed time since start: 0:32:39.832605.\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-599f950d8dc2>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m                                                    index_id)\n\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mindex_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwait_for_completion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_id\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Waiting for deployment'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-9-0dd7f532d61e>\u001b[0m in \u001b[0;36mwait_for_completion\u001b[0;34m(self, operation_id, message, sleep_time)\u001b[0m\n\u001b[1;32m     34\u001b[0m             logging.info('{}. Elapsed time since start: {}.'.format(\n\u001b[1;32m     35\u001b[0m                 message, str(elapsed_time)))\n\u001b[0;32m---> 36\u001b[0;31m             \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msleep_time\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'response'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "operation_id = deployment_client.create_deployment(deployment_display_name, \n",
    "                                                   deployed_index_id,\n",
    "                                                   endpoint_id,\n",
    "                                                   index_id)\n",
    "\n",
    "response = index_client.wait_for_completion(operation_id, 'Waiting for deployment', 10)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying the ANN service\n",
    "\n",
    "You will use the gRPC interface to query the deployed index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve the gRPC private endpoint for the ANN Match service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployed_index_ip = deployment_client.get_deployment_grpc_ip(endpoint_id, deployed_index_id)\n",
    "endpoint = f'{deployed_index_ip}:{MATCH_SERVICE_PORT}'\n",
    "print(f'gRPC endpoint for the: {deployed_index_id} deployment is: {endpoint}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a helper wrapper around the Match Service gRPC API.\n",
    "\n",
    "The wrapper uses the pre-generated gRPC stub to the Online Querying gRPC interface. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MatchService(object):\n",
    "    \"\"\"This is a wrapper around Online Querying gRPC interface.\"\"\"\n",
    "    def __init__(self, endpoint, deployed_index_id):\n",
    "        self.endpoint = endpoint\n",
    "        self.deployed_index_id = deployed_index_id\n",
    "\n",
    "    def single_match(\n",
    "        self,\n",
    "        embedding: List[float],\n",
    "        num_neighbors: int) -> List[Tuple[str, float]]:\n",
    "        \"\"\"Requests a match for a single embedding.\"\"\"\n",
    "    \n",
    "        match_request = match_pb2.MatchRequest(deployed_index_id=self.deployed_index_id,\n",
    "                                               float_val=embedding,\n",
    "                                               num_neighbors=num_neighbors)\n",
    "        with grpc.insecure_channel(endpoint) as channel:\n",
    "            stub = match_pb2_grpc.MatchServiceStub(channel)\n",
    "            response = stub.Match(match_request)\n",
    "    \n",
    "        return [(neighbor.id, neighbor.distance) for neighbor in response.neighbor]\n",
    "\n",
    "\n",
    "    def batch_match(\n",
    "        self,\n",
    "        embeddings: List[List[float]],\n",
    "        num_neighbors: int) -> List[List[Tuple[str, float]]]:\n",
    "        \"\"\"Requests matches ofr a list of embeddings.\"\"\"\n",
    "    \n",
    "        match_requests = [\n",
    "            match_pb2.MatchRequest(deployed_index_id=self.deployed_index_id,\n",
    "                                   float_val=embedding,\n",
    "                                   num_neighbors=num_neighbors)\n",
    "            for embedding in embeddings]\n",
    "    \n",
    "        batches_per_index = [\n",
    "            match_pb2.BatchMatchRequest.BatchMatchRequestPerIndex(\n",
    "                deployed_index_id=self.deployed_index_id,\n",
    "                requests=match_requests)]\n",
    "    \n",
    "        batch_match_request = match_pb2.BatchMatchRequest(\n",
    "            requests=batches_per_index)\n",
    "    \n",
    "        with grpc.insecure_channel(endpoint) as channel:\n",
    "            stub = match_pb2_grpc.MatchServiceStub(channel)\n",
    "            response = stub.BatchMatch(batch_match_request)\n",
    "        \n",
    "        matches = []\n",
    "        for batch_per_index in response.responses:\n",
    "            for match in batch_per_index.responses:\n",
    "                matches.append(\n",
    "                    [(neighbor.id, neighbor.distance) for neighbor in match.neighbor])\n",
    "        \n",
    "        return matches\n",
    "    \n",
    "match_service = MatchService(endpoint, deployed_index_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare sample data\n",
    "\n",
    "Retrieve a few embeddings from the BigQuery embedding table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bigquery df_embeddings\n",
    "\n",
    "SELECT id, embedding\n",
    "FROM `recommendations.item_embeddings` \n",
    "LIMIT 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_embeddings = [list(embedding) for embedding in df_embeddings['embedding']]\n",
    "sample_embeddings[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run a single match query\n",
    "\n",
    "The following call requests 10 closest neighbours for a single embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time \n",
    "\n",
    "single_match = match_service.single_match(sample_embeddings[0], 10)\n",
    "single_match"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run a batch match query\n",
    "\n",
    "The following call requests 3 closest neighbours for each of the embeddings in a batch of 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time \n",
    "\n",
    "batch_match = match_service.batch_match(sample_embeddings[0:5], 3)\n",
    "batch_match"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "\n",
    "**WARNING**\n",
    "\n",
    "The below code will delete all ANN deployments, endpoints, and indexes in the configured project.\n",
    "\n",
    "### Delete index deployments and endpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting endpoint: projects/895222332033/locations/us-central1/indexEndpoints/856246879153815552\n",
      "Deleting endpoint: projects/895222332033/locations/us-central1/indexEndpoints/3621457050359300096\n",
      "Deleting endpoint: projects/895222332033/locations/us-central1/indexEndpoints/108649341010313216\n"
     ]
    }
   ],
   "source": [
    "for endpoint in deployment_client.list_endpoints():\n",
    "    endpoint_id = endpoint['name'].split('/')[-1]\n",
    "    if 'deployedIndexes' in endpoint.keys():\n",
    "        for deployment in endpoint['deployedIndexes']:\n",
    "            print('   Deleting index deployment: {} in the endpoint: {} '.format(deployment['id'], endpoint_id))\n",
    "            deployment_client.delete_deployment(endpoint_id, deployment['id'])\n",
    "    print('Deleting endpoint: {}'.format(endpoint['name']))\n",
    "    deployment_client.delete_endpoint(endpoint_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete indexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleting index: projects/895222332033/locations/us-central1/indexes/1335880239468773376\n",
      "Deleting index: projects/895222332033/locations/us-central1/indexes/3544895856694001664\n",
      "Deleting index: projects/895222332033/locations/us-central1/indexes/4886968545650409472\n"
     ]
    }
   ],
   "source": [
    "for index in index_client.list_indexes():\n",
    "    index_id = index['name'].split('/')[-1]\n",
    "    print('Deleting index: {}'.format(index['name']))\n",
    "    index_client.delete_index(index_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License\n",
    "\n",
    "Copyright 2020 Google LLC\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License. You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. \n",
    "\n",
    "See the License for the specific language governing permissions and limitations under the License.\n",
    "\n",
    "**This is not an official Google product but sample code provided for an educational purpose**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-4.m61",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-4:m61"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}