self-paced-labs/vertex-ai/vertex-pipelines/tfx/lab_exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3c78dad1-bbc2-4f32-b4fa-ecc18285eacf",
   "metadata": {},
   "source": [
    "# Lab: Chicago taxifare tip prediction on Google Cloud Vertex Pipelines using the TFX SDK"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c0fc146-a03f-49eb-8a2b-92f97f1935de",
   "metadata": {},
   "source": [
    "## Learning objectives"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33ecae6b-f501-4dc2-8fa7-cc66f3176d00",
   "metadata": {},
   "source": [
    "* Define a machine learning pipeline to predict taxi fare tips using the TFX SDK.\n",
    "* Compile and run a TFX pipeline on Google Cloud's Vertex Pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f16d68ac-5d8c-4930-b622-45797462312f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a7eb3452-b06f-4617-8bb0-620526c016db",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "The [Chicago Taxi Trips](https://pantheon.corp.google.com/marketplace/details/city-of-chicago-public-data/chicago-taxi-trips) dataset is one of the [public datasets hosted with BigQuery](https://cloud.google.com/bigquery/public-data/), which includes taxi trips from 2013 to the present, reported to the City of Chicago in its role as a regulatory agency. The task is to predict whether a given trip will result in a tip > 20%."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19614574-410b-47e3-a249-dcc092156b44",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab6764e2-6f1c-4dc0-a770-579daf9ca5f5",
   "metadata": {},
   "source": [
    "### Define constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cc5267aa-5544-4d76-a835-1637f6f254ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "GOOGLE_CLOUD_PROJECT_ID = !(gcloud config get-value core/project)\n",
    "GOOGLE_CLOUD_PROJECT_ID = GOOGLE_CLOUD_PROJECT_ID[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fcac2822-fecb-4b8c-8808-6d1be78e7cf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "GOOGLE_CLOUD_REGION = 'us-central1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d656c11a-15a9-4b62-8eb0-bd03825aca78",
   "metadata": {},
   "outputs": [],
   "source": [
    "BQ_DATASET_NAME = 'chicago_taxifare_tips'\n",
    "BQ_TABLE_NAME = 'chicago_taxi_tips_ml'\n",
    "BQ_LOCATION = 'US'\n",
    "BQ_URI = f\"bq://{GOOGLE_CLOUD_PROJECT_ID}.{BQ_DATASET_NAME}.{BQ_TABLE_NAME}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "id": "61e3c5d0-4a46-4e99-839c-07e3059dc55b",
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_DISPLAY_NAME = 'chicago-taxifare-tips'\n",
    "MODEL_DISPLAY_NAME = f'{DATASET_DISPLAY_NAME}-classifier'\n",
    "PIPELINE_NAME = f'{MODEL_DISPLAY_NAME}-train-pipeline'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "565613f5-7b43-4158-ae6b-1836b8a89ac9",
   "metadata": {},
   "source": [
    "### Create Google Cloud Storage bucket for storing Vertex Pipeline artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3ba449d5-84fe-45bc-8bf0-10de640f9722",
   "metadata": {},
   "outputs": [],
   "source": [
    "GCS_LOCATION = f\"gs://{GOOGLE_CLOUD_PROJECT_ID}-tfx\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2041a651-22d0-4eaa-bb4b-aa4968268317",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://dougkelly-vertex-demos-tfx/...\n",
      "ServiceException: 409 A Cloud Storage bucket named 'dougkelly-vertex-demos-tfx' already exists. Try another name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.\n"
     ]
    }
   ],
   "source": [
    "!gsutil mb -l $GOOGLE_CLOUD_REGION $GCS_LOCATION"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be3d6ffc-9e50-4bea-a393-74625f022235",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "aef724ec-6963-46d3-afbf-32e4af5312d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tensorflow as tf\n",
    "import tfx\n",
    "import kfp\n",
    "\n",
    "from google.cloud import bigquery\n",
    "from google.cloud import aiplatform as vertex_ai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6c1166c8-4096-44d4-a6fa-b35d34c239e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow: 2.6.2\n",
      "tfx: 1.4.0\n",
      "kfp: 1.8.1\n",
      "Google Cloud Vertex AI Python SDK: 1.7.1\n"
     ]
    }
   ],
   "source": [
    "print(f\"tensorflow: {tf.__version__}\")\n",
    "print(f\"tfx: {tfx.__version__}\")\n",
    "print(f\"kfp: {kfp.__version__}\")\n",
    "print(f\"Google Cloud Vertex AI Python SDK: {vertex_ai.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eed174fa-206e-4a3d-80ea-2e6f1948106b",
   "metadata": {},
   "source": [
    "## Create BigQuery dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1c3f7c5a-d5bf-47c6-890b-c9f64f58c0d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset 'dougkelly-vertex-demos:chicago_taxifare_tips' successfully created.\n"
     ]
    }
   ],
   "source": [
    "!bq --location=$BQ_LOCATION mk -d \\\n",
    "$GOOGLE_CLOUD_PROJECT_ID:$BQ_DATASET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e627a7f-027d-422b-a3b4-880ab26ad56d",
   "metadata": {},
   "source": [
    "## Create BigQuery dataset for ML classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "52be50bf-de33-446f-8e83-05e31005ae34",
   "metadata": {},
   "outputs": [],
   "source": [
    "SAMPLE_SIZE = 20000\n",
    "YEAR = 2020"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "1b1f3be3-d8dd-458b-92d8-8b01b8791ede",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_script = '''\n",
    "CREATE OR REPLACE TABLE `@PROJECT_ID.@DATASET.@TABLE` \n",
    "AS (\n",
    "    WITH\n",
    "      taxitrips AS (\n",
    "      SELECT\n",
    "        trip_start_timestamp,\n",
    "        trip_seconds,\n",
    "        trip_miles,\n",
    "        payment_type,\n",
    "        pickup_longitude,\n",
    "        pickup_latitude,\n",
    "        dropoff_longitude,\n",
    "        dropoff_latitude,\n",
    "        tips,\n",
    "        fare\n",
    "      FROM\n",
    "        `bigquery-public-data.chicago_taxi_trips.taxi_trips`\n",
    "      WHERE 1=1 \n",
    "      AND pickup_longitude IS NOT NULL\n",
    "      AND pickup_latitude IS NOT NULL\n",
    "      AND dropoff_longitude IS NOT NULL\n",
    "      AND dropoff_latitude IS NOT NULL\n",
    "      AND trip_miles > 0\n",
    "      AND trip_seconds > 0\n",
    "      AND fare > 0\n",
    "      AND EXTRACT(YEAR FROM trip_start_timestamp) = @YEAR\n",
    "    )\n",
    "\n",
    "    SELECT\n",
    "      trip_start_timestamp,\n",
    "      EXTRACT(MONTH from trip_start_timestamp) as trip_month,\n",
    "      EXTRACT(DAY from trip_start_timestamp) as trip_day,\n",
    "      EXTRACT(DAYOFWEEK from trip_start_timestamp) as trip_day_of_week,\n",
    "      EXTRACT(HOUR from trip_start_timestamp) as trip_hour,\n",
    "      trip_seconds,\n",
    "      trip_miles,\n",
    "      payment_type,\n",
    "      ST_AsText(\n",
    "          ST_SnapToGrid(ST_GeogPoint(pickup_longitude, pickup_latitude), 0.1)\n",
    "      ) AS pickup_grid,\n",
    "      ST_AsText(\n",
    "          ST_SnapToGrid(ST_GeogPoint(dropoff_longitude, dropoff_latitude), 0.1)\n",
    "      ) AS dropoff_grid,\n",
    "      ST_Distance(\n",
    "          ST_GeogPoint(pickup_longitude, pickup_latitude), \n",
    "          ST_GeogPoint(dropoff_longitude, dropoff_latitude)\n",
    "      ) AS euclidean,\n",
    "      CONCAT(\n",
    "          ST_AsText(ST_SnapToGrid(ST_GeogPoint(pickup_longitude,\n",
    "              pickup_latitude), 0.1)), \n",
    "          ST_AsText(ST_SnapToGrid(ST_GeogPoint(dropoff_longitude,\n",
    "              dropoff_latitude), 0.1))\n",
    "      ) AS loc_cross,\n",
    "      IF((tips/fare >= 0.2), 1, 0) AS tip_bin,\n",
    "      IF(ABS(MOD(FARM_FINGERPRINT(STRING(trip_start_timestamp)), 10)) < 9, 'UNASSIGNED', 'TEST') AS ml_use\n",
    "    FROM\n",
    "      taxitrips\n",
    "    LIMIT @LIMIT\n",
    ")\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "2780c13f-4d9d-462d-b370-e753d1f596bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "sql_script = sql_script.replace(\n",
    "    '@PROJECT_ID', GOOGLE_CLOUD_PROJECT_ID).replace(\n",
    "    '@DATASET', BQ_DATASET_NAME).replace(\n",
    "    '@TABLE', BQ_TABLE_NAME).replace(\n",
    "    '@YEAR', str(YEAR)).replace(\n",
    "    '@LIMIT', str(SAMPLE_SIZE))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "6e1a1755-1cc9-4f8e-9b70-a0c8384e843e",
   "metadata": {},
   "outputs": [],
   "source": [
    "bq_client = bigquery.Client(project=GOOGLE_CLOUD_PROJECT_ID, location=BQ_LOCATION)\n",
    "job = bq_client.query(sql_script)\n",
    "_ = job.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "cc7656de-762f-4d91-aa61-526737d8d177",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Query complete after 0.00s: 100%|██████████| 2/2 [00:00<00:00, 1100.00query/s]                        \n",
      "Downloading: 100%|██████████| 2/2 [00:01<00:00,  1.49rows/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ml_use</th>\n",
       "      <th>f0_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>UNASSIGNED</td>\n",
       "      <td>18127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TEST</td>\n",
       "      <td>1873</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       ml_use    f0_\n",
       "0  UNASSIGNED  18127\n",
       "1        TEST   1873"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%bigquery\n",
    "\n",
    "SELECT ml_use, COUNT(*)\n",
    "FROM chicago_taxifare_tips.chicago_taxi_tips_ml\n",
    "GROUP BY ml_use"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54dcce56-6ef5-49fe-a2f8-9556fc96a119",
   "metadata": {},
   "source": [
    "## Create a Vertex AI managed dataset resource for pipeline dataset lineage tracking"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c7b5043-2be4-437b-965e-5b96853a340b",
   "metadata": {},
   "source": [
    "### Initialize Vertex AI Python SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "94bb4d8c-f454-454d-b7eb-120dfea14280",
   "metadata": {},
   "outputs": [],
   "source": [
    "vertex_ai.init(project=GOOGLE_CLOUD_PROJECT_ID, location=GOOGLE_CLOUD_REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "666a9708-40ca-431f-84b1-6a89e29f90d2",
   "metadata": {},
   "source": [
    "### Create Vertex managed tabular dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "31b9ff56-26c4-4449-98ed-7e16ced14487",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.datasets.dataset:Creating TabularDataset\n",
      "INFO:google.cloud.aiplatform.datasets.dataset:Create TabularDataset backing LRO: projects/617979904441/locations/us-central1/datasets/4914403559786676224/operations/8192561262836580352\n",
      "INFO:google.cloud.aiplatform.datasets.dataset:TabularDataset created. Resource name: projects/617979904441/locations/us-central1/datasets/4914403559786676224\n",
      "INFO:google.cloud.aiplatform.datasets.dataset:To use this TabularDataset in another session:\n",
      "INFO:google.cloud.aiplatform.datasets.dataset:ds = aiplatform.TabularDataset('projects/617979904441/locations/us-central1/datasets/4914403559786676224')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "name: \"projects/617979904441/locations/us-central1/datasets/4914403559786676224\"\n",
       "display_name: \"chicago-taxifare-tips\"\n",
       "metadata_schema_uri: \"gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml\"\n",
       "create_time {\n",
       "  seconds: 1638842506\n",
       "  nanos: 675382000\n",
       "}\n",
       "update_time {\n",
       "  seconds: 1638842507\n",
       "  nanos: 274500000\n",
       "}\n",
       "etag: \"AMEw9yMUktBa4vLbkjJG08QJYd65MitZkjzr9qSHPVOlHLazO097R4ZOem8WwjyM1psz\"\n",
       "labels {\n",
       "  key: \"aiplatform.googleapis.com/dataset_metadata_schema\"\n",
       "  value: \"TABLE\"\n",
       "}\n",
       "metadata {\n",
       "  struct_value {\n",
       "    fields {\n",
       "      key: \"inputConfig\"\n",
       "      value {\n",
       "        struct_value {\n",
       "          fields {\n",
       "            key: \"bigquerySource\"\n",
       "            value {\n",
       "              struct_value {\n",
       "                fields {\n",
       "                  key: \"uri\"\n",
       "                  value {\n",
       "                    string_value: \"bq://dougkelly-vertex-demos.chicago_taxifare_tips.chicago_taxi_tips_ml\"\n",
       "                  }\n",
       "                }\n",
       "              }\n",
       "            }\n",
       "          }\n",
       "        }\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "}"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tabular_dataset = vertex_ai.TabularDataset.create(display_name=f\"{DATASET_DISPLAY_NAME}\", bq_source=f\"{BQ_URI}\")\n",
    "tabular_dataset.gca_resource"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f53417c-65e2-46d2-94eb-46936cc61fb9",
   "metadata": {},
   "source": [
    "## Create a TFX pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "17dd8442-12a5-4eef-880f-afa8ebcdb414",
   "metadata": {},
   "outputs": [],
   "source": [
    "PIPELINE_DIR=\"tfx_taxifare_tips\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e843dfd9-0f10-462a-bb05-004ee4be0f20",
   "metadata": {},
   "source": [
    "### Write model code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "867de278-6b20-4844-87b3-801105b71e9d",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c2562a4-435c-49a3-825a-9a5435e97c52",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile {PIPELINE_DIR}/model_training/features.py\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94fe59a2-7064-4bb2-b4b1-2206464c6e83",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile {PIPELINE_DIR}/model_training/preprocessing.py\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b9426ff-470b-4960-96c5-cb016743f487",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile {PIPELINE_DIR}/model_training/model.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dfe0130-731a-44a6-a883-b6830bf295a7",
   "metadata": {},
   "source": [
    "### Write pipeline definition with the TFX SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f176f69-9233-4cbd-a1b9-27440cc7fc36",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2713bdd-5262-4224-965f-e770a6bc1d85",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile {PIPELINE_DIR}/pipeline.py\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3a38825-409b-48e3-bf78-47b97fa117f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile {PIPELINE_DIR}/runner.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a56673d8-9072-4cd9-94f0-ede8a88074ca",
   "metadata": {},
   "source": [
    "## Run your TFX pipeline on Vertex Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e814f703-5de0-4dcb-9263-ff6ba2eae877",
   "metadata": {},
   "source": [
    "### Create a Artifact Registry on Google Cloud for your pipeline container image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "3291170d-90ea-42e2-a508-f41ab4d5c026",
   "metadata": {},
   "outputs": [],
   "source": [
    "ARTIFACT_REGISTRY=\"tfx-taxifare-tips\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "93175bc2-4ffd-4944-96fc-7be4f4cca0cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;31mERROR:\u001b[0m (gcloud.artifacts.repositories.create) ALREADY_EXISTS: the repository already exists\n"
     ]
    }
   ],
   "source": [
    "# TODO: create a Docker Artifact Registry using the gcloud CLI.\n",
    "# Documentation link: https://cloud.google.com/sdk/gcloud/reference/artifacts/repositories/create\n",
    "\n",
    "!gcloud artifacts repositories create {ARTIFACT_REGISTRY} \\\n",
    "--repository-format=docker \\\n",
    "--location={GOOGLE_CLOUD_REGION} \\\n",
    "--description=\"Artifact registry for TFX pipeline images for Chicago taxifare prediction.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "89e7f6f0-c9c0-4c05-a282-6b7e7ea2ad04",
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGE_NAME=\"tfx-taxifare-tips\"\n",
    "IMAGE_TAG=\"latest\"\n",
    "IMAGE_URI=f\"{GOOGLE_CLOUD_REGION}-docker.pkg.dev/{GOOGLE_CLOUD_PROJECT_ID}/{ARTIFACT_REGISTRY}/{IMAGE_NAME}:{IMAGE_TAG}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f5e5d41-0486-4ace-a4fc-0f8dae7fb95d",
   "metadata": {},
   "source": [
    "### Set the pipeline configurations for the Vertex AI run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "id": "067d386a-b5c9-4f57-afeb-ca2426cb865c",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ[\"DATASET_DISPLAY_NAME\"] = DATASET_DISPLAY_NAME\n",
    "os.environ[\"MODEL_DISPLAY_NAME\"] = MODEL_DISPLAY_NAME\n",
    "os.environ[\"PIPELINE_NAME\"] = PIPELINE_NAME\n",
    "os.environ[\"GOOGLE_CLOUD_PROJECT_ID\"] = GOOGLE_CLOUD_PROJECT_ID\n",
    "os.environ[\"GOOGLE_CLOUD_REGION\"] = GOOGLE_CLOUD_REGION\n",
    "os.environ[\"GCS_LOCATION\"] = GCS_LOCATION\n",
    "os.environ[\"TRAIN_LIMIT\"] = \"5000\"\n",
    "os.environ[\"TEST_LIMIT\"] = \"1000\"\n",
    "os.environ[\"BEAM_RUNNER\"] = \"DataflowRunner\"\n",
    "os.environ[\"TRAINING_RUNNER\"] = \"vertex\"\n",
    "os.environ[\"TFX_IMAGE_URI\"] = IMAGE_URI\n",
    "os.environ[\"ENABLE_CACHE\"] = \"1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "id": "b7d86b95-693c-4506-8885-3872ea07184b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GOOGLE_CLOUD_PROJECT_ID: dougkelly-vertex-demos\n",
      "GOOGLE_CLOUD_REGION: us-central1\n",
      "GCS_LOCATION: gs://dougkelly-vertex-demos-tfx\n",
      "ARTIFACT_STORE_URI: gs://dougkelly-vertex-demos-tfx/tfx-artifacts\n",
      "MODEL_REGISTRY_URI: gs://dougkelly-vertex-demos-tfx/model-registry\n",
      "DATASET_DISPLAY_NAME: chicago-taxifare-tips\n",
      "MODEL_DISPLAY_NAME: chicago-taxifare-tips-classifier\n",
      "PIPELINE_NAME: chicago-taxifare-tips-classifier-train-pipeline\n",
      "ML_USE_COLUMN: ml_use\n",
      "EXCLUDE_COLUMNS: trip_start_timestamp\n",
      "TRAIN_LIMIT: 5000\n",
      "TEST_LIMIT: 1000\n",
      "SERVE_LIMIT: 0\n",
      "NUM_TRAIN_SPLITS: 4\n",
      "NUM_EVAL_SPLITS: 1\n",
      "ACCURACY_THRESHOLD: 0.8\n",
      "USE_KFP_SA: False\n",
      "TFX_IMAGE_URI: us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips:latest\n",
      "BEAM_RUNNER: DataflowRunner\n",
      "BEAM_DIRECT_PIPELINE_ARGS: ['--project=dougkelly-vertex-demos', '--temp_location=gs://dougkelly-vertex-demos-tfx/temp']\n",
      "BEAM_DATAFLOW_PIPELINE_ARGS: ['--project=dougkelly-vertex-demos', '--temp_location=gs://dougkelly-vertex-demos-tfx/temp', '--region=us-central1', '--runner=DataflowRunner']\n",
      "TRAINING_RUNNER: vertex\n",
      "VERTEX_TRAINING_ARGS: {'project': 'dougkelly-vertex-demos', 'worker_pool_specs': [{'machine_spec': {'machine_type': 'n1-standard-4'}, 'replica_count': 1, 'container_spec': {'image_uri': 'us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips:latest'}}]}\n",
      "VERTEX_TRAINING_CONFIG: {'ai_platform_training_enable_ucaip': True, 'ai_platform_training_ucaip_region': 'us-central1', 'ai_platform_training_args': {'project': 'dougkelly-vertex-demos', 'worker_pool_specs': [{'machine_spec': {'machine_type': 'n1-standard-4'}, 'replica_count': 1, 'container_spec': {'image_uri': 'us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips:latest'}}]}, 'use_gpu': False}\n",
      "ENABLE_CACHE: 1\n"
     ]
    }
   ],
   "source": [
    "from tfx_taxifare_tips.tfx_pipeline import config\n",
    "import importlib\n",
    "importlib.reload(config)\n",
    "\n",
    "for key, value in config.__dict__.items():\n",
    "    if key.isupper(): print(f'{key}: {value}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3958359-abdc-4cec-a435-7c775d05c3fb",
   "metadata": {},
   "source": [
    "### Build the TFX pipeline container image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "id": "4b50cdf5-cc4b-4352-839f-a48a04315361",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips:latest\n"
     ]
    }
   ],
   "source": [
    "!echo $TFX_IMAGE_URI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "id": "860a5559-ea0f-48b3-94f1-871ec084e35d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !docker build . -t test-image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "id": "9f2adad6-ff8e-41be-9d56-ec078822e197",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating temporary tarball archive of 26 file(s) totalling 267.1 KiB before compression.\n",
      "Uploading tarball of [.] to [gs://dougkelly-vertex-demos_cloudbuild/source/1639112263.267746-dc49aada98f941ca8a3d522f7e57a263.tgz]\n",
      "Created [https://cloudbuild.googleapis.com/v1/projects/dougkelly-vertex-demos/locations/global/builds/2b8a42f1-fa91-43ee-b1ac-3269318afa84].\n",
      "Logs are available at [https://console.cloud.google.com/cloud-build/builds/2b8a42f1-fa91-43ee-b1ac-3269318afa84?project=617979904441].\n",
      "----------------------------- REMOTE BUILD OUTPUT ------------------------------\n",
      "starting build \"2b8a42f1-fa91-43ee-b1ac-3269318afa84\"\n",
      "\n",
      "FETCHSOURCE\n",
      "Fetching storage object: gs://dougkelly-vertex-demos_cloudbuild/source/1639112263.267746-dc49aada98f941ca8a3d522f7e57a263.tgz#1639112263639872\n",
      "Copying gs://dougkelly-vertex-demos_cloudbuild/source/1639112263.267746-dc49aada98f941ca8a3d522f7e57a263.tgz#1639112263639872...\n",
      "/ [1 files][ 48.4 KiB/ 48.4 KiB]                                                \n",
      "Operation completed over 1 objects/48.4 KiB.                                     \n",
      "BUILD\n",
      "Already have image (with digest): gcr.io/cloud-builders/docker\n",
      "Sending build context to Docker daemon  297.5kB\n",
      "Step 1/5 : FROM gcr.io/tfx-oss-public/tfx:1.4.0\n",
      "1.4.0: Pulling from tfx-oss-public/tfx\n",
      "e4ca327ec0e7: Pulling fs layer\n",
      "0fa9fc055636: Pulling fs layer\n",
      "448bb2d7fba5: Pulling fs layer\n",
      "a084e2627368: Pulling fs layer\n",
      "de932d3a14d8: Pulling fs layer\n",
      "ebe7db8e97e0: Pulling fs layer\n",
      "66fef8aabad3: Pulling fs layer\n",
      "9696f5331161: Pulling fs layer\n",
      "7799e2177407: Pulling fs layer\n",
      "56d35ebee226: Pulling fs layer\n",
      "f23d5847df7e: Pulling fs layer\n",
      "85729b3aa914: Pulling fs layer\n",
      "27594cd25c9b: Pulling fs layer\n",
      "420dbd17143e: Pulling fs layer\n",
      "71797a45a0de: Pulling fs layer\n",
      "b073ad8b8421: Pulling fs layer\n",
      "4b7516634a4b: Pulling fs layer\n",
      "434af55afd20: Pulling fs layer\n",
      "85d42009e725: Pulling fs layer\n",
      "a084e2627368: Waiting\n",
      "de932d3a14d8: Waiting\n",
      "6d895d7b587a: Pulling fs layer\n",
      "9ff76d709282: Pulling fs layer\n",
      "ebe7db8e97e0: Waiting\n",
      "4160ec915caf: Pulling fs layer\n",
      "66fef8aabad3: Waiting\n",
      "eb84f1e37919: Pulling fs layer\n",
      "b073ad8b8421: Waiting\n",
      "85729b3aa914: Waiting\n",
      "a616daa88ca5: Pulling fs layer\n",
      "9696f5331161: Waiting\n",
      "27594cd25c9b: Waiting\n",
      "420dbd17143e: Waiting\n",
      "71797a45a0de: Waiting\n",
      "99b8f7435830: Pulling fs layer\n",
      "1e2f495eab09: Pulling fs layer\n",
      "d880b09bf9ec: Pulling fs layer\n",
      "56d35ebee226: Waiting\n",
      "f23d5847df7e: Waiting\n",
      "434af55afd20: Waiting\n",
      "85d42009e725: Waiting\n",
      "6d895d7b587a: Waiting\n",
      "eb84f1e37919: Waiting\n",
      "9ff76d709282: Waiting\n",
      "4160ec915caf: Waiting\n",
      "a616daa88ca5: Waiting\n",
      "4b7516634a4b: Waiting\n",
      "99b8f7435830: Waiting\n",
      "1e2f495eab09: Waiting\n",
      "7799e2177407: Waiting\n",
      "403fe241a00b: Pulling fs layer\n",
      "ca533ed5e802: Pulling fs layer\n",
      "9d118b4c5fa6: Pulling fs layer\n",
      "388915f48178: Pulling fs layer\n",
      "d880b09bf9ec: Waiting\n",
      "ddeefb06ab95: Pulling fs layer\n",
      "62d0755e4559: Pulling fs layer\n",
      "58a229d727c0: Pulling fs layer\n",
      "c4606a99c9de: Pulling fs layer\n",
      "403fe241a00b: Waiting\n",
      "ca533ed5e802: Waiting\n",
      "62d0755e4559: Waiting\n",
      "9d118b4c5fa6: Waiting\n",
      "58a229d727c0: Waiting\n",
      "388915f48178: Waiting\n",
      "ddeefb06ab95: Waiting\n",
      "448bb2d7fba5: Download complete\n",
      "0fa9fc055636: Verifying Checksum\n",
      "0fa9fc055636: Download complete\n",
      "e4ca327ec0e7: Verifying Checksum\n",
      "e4ca327ec0e7: Download complete\n",
      "a084e2627368: Verifying Checksum\n",
      "a084e2627368: Download complete\n",
      "de932d3a14d8: Download complete\n",
      "66fef8aabad3: Download complete\n",
      "7799e2177407: Verifying Checksum\n",
      "7799e2177407: Download complete\n",
      "e4ca327ec0e7: Pull complete\n",
      "0fa9fc055636: Pull complete\n",
      "448bb2d7fba5: Pull complete\n",
      "a084e2627368: Pull complete\n",
      "de932d3a14d8: Pull complete\n",
      "9696f5331161: Verifying Checksum\n",
      "9696f5331161: Download complete\n",
      "f23d5847df7e: Verifying Checksum\n",
      "f23d5847df7e: Download complete\n",
      "ebe7db8e97e0: Verifying Checksum\n",
      "ebe7db8e97e0: Download complete\n",
      "56d35ebee226: Verifying Checksum\n",
      "56d35ebee226: Download complete\n",
      "420dbd17143e: Verifying Checksum\n",
      "420dbd17143e: Download complete\n",
      "71797a45a0de: Verifying Checksum\n",
      "71797a45a0de: Download complete\n",
      "27594cd25c9b: Verifying Checksum\n",
      "27594cd25c9b: Download complete\n",
      "4b7516634a4b: Verifying Checksum\n",
      "4b7516634a4b: Download complete\n",
      "434af55afd20: Download complete\n",
      "85d42009e725: Verifying Checksum\n",
      "85d42009e725: Download complete\n",
      "6d895d7b587a: Verifying Checksum\n",
      "6d895d7b587a: Download complete\n",
      "9ff76d709282: Verifying Checksum\n",
      "9ff76d709282: Download complete\n",
      "4160ec915caf: Verifying Checksum\n",
      "4160ec915caf: Download complete\n",
      "eb84f1e37919: Verifying Checksum\n",
      "eb84f1e37919: Download complete\n",
      "a616daa88ca5: Verifying Checksum\n",
      "a616daa88ca5: Download complete\n",
      "99b8f7435830: Download complete\n",
      "85729b3aa914: Verifying Checksum\n",
      "85729b3aa914: Download complete\n",
      "b073ad8b8421: Verifying Checksum\n",
      "b073ad8b8421: Download complete\n",
      "403fe241a00b: Verifying Checksum\n",
      "403fe241a00b: Download complete\n",
      "1e2f495eab09: Download complete\n",
      "9d118b4c5fa6: Verifying Checksum\n",
      "9d118b4c5fa6: Download complete\n",
      "388915f48178: Verifying Checksum\n",
      "388915f48178: Download complete\n",
      "ddeefb06ab95: Verifying Checksum\n",
      "ddeefb06ab95: Download complete\n",
      "62d0755e4559: Verifying Checksum\n",
      "62d0755e4559: Download complete\n",
      "58a229d727c0: Verifying Checksum\n",
      "58a229d727c0: Download complete\n",
      "ca533ed5e802: Download complete\n",
      "c4606a99c9de: Verifying Checksum\n",
      "c4606a99c9de: Download complete\n",
      "d880b09bf9ec: Verifying Checksum\n",
      "d880b09bf9ec: Download complete\n",
      "ebe7db8e97e0: Pull complete\n",
      "66fef8aabad3: Pull complete\n",
      "9696f5331161: Pull complete\n",
      "7799e2177407: Pull complete\n",
      "56d35ebee226: Pull complete\n",
      "f23d5847df7e: Pull complete\n",
      "85729b3aa914: Pull complete\n",
      "27594cd25c9b: Pull complete\n",
      "420dbd17143e: Pull complete\n",
      "71797a45a0de: Pull complete\n",
      "b073ad8b8421: Pull complete\n",
      "4b7516634a4b: Pull complete\n",
      "434af55afd20: Pull complete\n",
      "85d42009e725: Pull complete\n",
      "6d895d7b587a: Pull complete\n",
      "9ff76d709282: Pull complete\n",
      "4160ec915caf: Pull complete\n",
      "eb84f1e37919: Pull complete\n",
      "a616daa88ca5: Pull complete\n",
      "99b8f7435830: Pull complete\n",
      "1e2f495eab09: Pull complete\n",
      "d880b09bf9ec: Pull complete\n",
      "403fe241a00b: Pull complete\n",
      "ca533ed5e802: Pull complete\n",
      "9d118b4c5fa6: Pull complete\n",
      "388915f48178: Pull complete\n",
      "ddeefb06ab95: Pull complete\n",
      "62d0755e4559: Pull complete\n",
      "58a229d727c0: Pull complete\n",
      "c4606a99c9de: Pull complete\n",
      "Digest: sha256:1c90d7c7df1d78147013ae8d0377b1496b211391d740b88da6d9892946c30fb1\n",
      "Status: Downloaded newer image for gcr.io/tfx-oss-public/tfx:1.4.0\n",
      " ---> 8badc9bc28b6\n",
      "Step 2/5 : COPY requirements.txt requirements.txt\n",
      " ---> c6ea2cfbfb34\n",
      "Step 3/5 : RUN pip install -r requirements.txt\n",
      " ---> Running in 3c0db03dcad5\n",
      "Requirement already satisfied: tfx==1.4.0 in /opt/conda/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (1.4.0)\n",
      "Collecting kfp==1.8.1\n",
      "  Downloading kfp-1.8.1.tar.gz (248 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting google-cloud-aiplatform==1.7.1\n",
      "  Downloading google_cloud_aiplatform-1.7.1-py2.py3-none-any.whl (1.6 MB)\n",
      "Requirement already satisfied: grpcio<2,>=1.28.1 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.41.1)\n",
      "Requirement already satisfied: google-api-python-client<2,>=1.8 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.12.8)\n",
      "Requirement already satisfied: tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (2.6.2)\n",
      "Requirement already satisfied: numpy<1.20,>=1.16 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.19.5)\n",
      "Requirement already satisfied: pyyaml<6,>=3.12 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (5.4.1)\n",
      "Requirement already satisfied: tensorflow-hub<0.13,>=0.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (0.12.0)\n",
      "Requirement already satisfied: ml-metadata<1.5.0,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.4.0)\n",
      "Requirement already satisfied: kubernetes<13,>=10.0.1 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (12.0.1)\n",
      "Requirement already satisfied: ml-pipelines-sdk==1.4.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.4.0)\n",
      "Requirement already satisfied: tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<3,>=1.15 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (2.6.1)\n",
      "Requirement already satisfied: absl-py<0.13,>=0.9 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (0.12.0)\n",
      "Requirement already satisfied: tfx-bsl<1.5.0,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.4.0)\n",
      "Requirement already satisfied: tensorflow-transform<1.5.0,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.4.0)\n",
      "Requirement already satisfied: keras-tuner<2,>=1.0.4 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.0.4)\n",
      "Requirement already satisfied: tensorflow-data-validation<1.5.0,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.4.0)\n",
      "Requirement already satisfied: jinja2<4,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (2.11.3)\n",
      "Requirement already satisfied: click<8,>=7 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (7.1.2)\n",
      "Requirement already satisfied: google-cloud-bigquery<3,>=2.26.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (2.30.1)\n",
      "Requirement already satisfied: portpicker<2,>=1.3.1 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (1.5.0)\n",
      "Requirement already satisfied: protobuf<4,>=3.13 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (3.19.1)\n",
      "Requirement already satisfied: attrs<21,>=19.3.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (20.3.0)\n",
      "Requirement already satisfied: google-apitools<1,>=0.5 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (0.5.31)\n",
      "Requirement already satisfied: docker<5,>=4.1 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (4.4.4)\n",
      "Requirement already satisfied: apache-beam[gcp]<3,>=2.33 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (2.33.0)\n",
      "Requirement already satisfied: tensorflow-model-analysis<0.36,>=0.35.0 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (0.35.0)\n",
      "Requirement already satisfied: pyarrow<6,>=1 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (4.0.1)\n",
      "Requirement already satisfied: packaging<21,>=20 in /opt/conda/lib/python3.7/site-packages (from tfx==1.4.0->-r requirements.txt (line 1)) (20.9)\n",
      "Collecting absl-py<0.13,>=0.9\n",
      "  Downloading absl_py-0.11.0-py3-none-any.whl (127 kB)\n",
      "Requirement already satisfied: google-cloud-storage<2,>=1.20.0 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.1->-r requirements.txt (line 2)) (1.42.3)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.1 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.1->-r requirements.txt (line 2)) (1.35.0)\n",
      "Collecting requests-toolbelt<1,>=0.8.0\n",
      "  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)\n",
      "Collecting cloudpickle<2,>=1.3.0\n",
      "  Downloading cloudpickle-1.6.0-py3-none-any.whl (23 kB)\n",
      "Collecting kfp-server-api<2.0.0,>=1.1.2\n",
      "  Downloading kfp-server-api-1.7.1.tar.gz (52 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting jsonschema<4,>=3.0.1\n",
      "  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)\n",
      "Collecting tabulate<1,>=0.8.6\n",
      "  Downloading tabulate-0.8.9-py3-none-any.whl (25 kB)\n",
      "Collecting Deprecated<2,>=1.2.7\n",
      "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
      "Collecting strip-hints<1,>=0.1.8\n",
      "  Downloading strip-hints-0.1.10.tar.gz (29 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting docstring-parser<1,>=0.7.3\n",
      "  Downloading docstring_parser-0.13.tar.gz (23 kB)\n",
      "  Installing build dependencies: started\n",
      "  Installing build dependencies: finished with status 'done'\n",
      "  Getting requirements to build wheel: started\n",
      "  Getting requirements to build wheel: finished with status 'done'\n",
      "  Preparing metadata (pyproject.toml): started\n",
      "  Preparing metadata (pyproject.toml): finished with status 'done'\n",
      "Requirement already satisfied: kfp-pipeline-spec<0.2.0,>=0.1.10 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.1->-r requirements.txt (line 2)) (0.1.13)\n",
      "Collecting fire<1,>=0.3.1\n",
      "  Downloading fire-0.4.0.tar.gz (87 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Requirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.1->-r requirements.txt (line 2)) (1.8.2)\n",
      "Requirement already satisfied: google-api-core[grpc]<3.0.0dev,>=1.26.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==1.7.1->-r requirements.txt (line 3)) (1.31.4)\n",
      "Requirement already satisfied: proto-plus>=1.10.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==1.7.1->-r requirements.txt (line 3)) (1.19.7)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<0.13,>=0.9->tfx==1.4.0->-r requirements.txt (line 1)) (1.15.0)\n",
      "Requirement already satisfied: pytz>=2018.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (2021.3)\n",
      "Requirement already satisfied: avro-python3!=1.9.2,<1.10.0,>=1.8.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.9.2.1)\n",
      "Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (3.12.1)\n",
      "Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (2.6.0)\n",
      "Requirement already satisfied: orjson<4.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (3.6.4)\n",
      "Requirement already satisfied: oauth2client<5,>=2.0.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (4.1.3)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.24.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (2.25.1)\n",
      "Requirement already satisfied: pydot<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.4.2)\n",
      "Requirement already satisfied: typing-extensions<4,>=3.7.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (3.7.4.3)\n",
      "Requirement already satisfied: future<1.0.0,>=0.18.2 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (0.18.2)\n",
      "Requirement already satisfied: crcmod<2.0,>=1.7 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.7)\n",
      "Requirement already satisfied: fastavro<2,>=0.21.4 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.4.7)\n",
      "Requirement already satisfied: httplib2<0.20.0,>=0.8 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (0.19.1)\n",
      "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (2.8.2)\n",
      "Requirement already satisfied: dill<0.3.2,>=0.3.1.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (0.3.1.1)\n",
      "Requirement already satisfied: google-cloud-vision<2,>=0.38.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.0.0)\n",
      "Requirement already satisfied: google-cloud-bigtable<2,>=0.31.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.7.0)\n",
      "Requirement already satisfied: google-cloud-datastore<2,>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.15.3)\n",
      "Requirement already satisfied: google-cloud-spanner<2,>=1.13.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.19.1)\n",
      "Requirement already satisfied: google-cloud-dlp<2,>=0.12.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.0.0)\n",
      "Requirement already satisfied: grpcio-gcp<1,>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (0.2.2)\n",
      "Requirement already satisfied: google-cloud-recommendations-ai<=0.2.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (0.2.0)\n",
      "Requirement already satisfied: cachetools<5,>=3.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (4.2.4)\n",
      "Requirement already satisfied: google-cloud-core<2,>=0.28.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.7.2)\n",
      "Requirement already satisfied: google-cloud-videointelligence<2,>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.16.1)\n",
      "Requirement already satisfied: google-cloud-language<2,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.3.0)\n",
      "Requirement already satisfied: google-cloud-pubsub<2,>=0.39.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (1.7.0)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.7/site-packages (from Deprecated<2,>=1.2.7->kfp==1.8.1->-r requirements.txt (line 2)) (1.12.1)\n",
      "Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker<5,>=4.1->tfx==1.4.0->-r requirements.txt (line 1)) (0.57.0)\n",
      "Requirement already satisfied: termcolor in /opt/conda/lib/python3.7/site-packages (from fire<1,>=0.3.1->kfp==1.8.1->-r requirements.txt (line 2)) (1.1.0)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.1->-r requirements.txt (line 3)) (58.5.3)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.1->-r requirements.txt (line 3)) (1.53.0)\n",
      "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx==1.4.0->-r requirements.txt (line 1)) (3.0.1)\n",
      "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx==1.4.0->-r requirements.txt (line 1)) (0.1.0)\n",
      "Requirement already satisfied: fasteners>=0.14 in /opt/conda/lib/python3.7/site-packages (from google-apitools<1,>=0.5->tfx==1.4.0->-r requirements.txt (line 1)) (0.16.3)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp==1.8.1->-r requirements.txt (line 2)) (0.2.7)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp==1.8.1->-r requirements.txt (line 2)) (4.7.2)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3,>=2.26.0->tfx==1.4.0->-r requirements.txt (line 1)) (2.1.0)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from jinja2<4,>=2.7.3->tfx==1.4.0->-r requirements.txt (line 1)) (1.1.1)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp==1.8.1->-r requirements.txt (line 2)) (0.18.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp==1.8.1->-r requirements.txt (line 2)) (4.8.1)\n",
      "Requirement already satisfied: tensorboard in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (2.6.0)\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (1.7.1)\n",
      "Requirement already satisfied: kt-legacy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (1.0.4)\n",
      "Requirement already satisfied: ipython in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (7.29.0)\n",
      "Requirement already satisfied: urllib3>=1.15 in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp==1.8.1->-r requirements.txt (line 2)) (1.26.7)\n",
      "Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp==1.8.1->-r requirements.txt (line 2)) (2021.10.8)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx==1.4.0->-r requirements.txt (line 1)) (1.3.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging<21,>=20->tfx==1.4.0->-r requirements.txt (line 1)) (2.4.7)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from portpicker<2,>=1.3.1->tfx==1.4.0->-r requirements.txt (line 1)) (5.8.0)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.7/site-packages (from strip-hints<1,>=0.1.8->kfp==1.8.1->-r requirements.txt (line 2)) (0.37.0)\n",
      "Requirement already satisfied: gast==0.4.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (0.4.0)\n",
      "Requirement already satisfied: flatbuffers~=1.12.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (1.12)\n",
      "Requirement already satisfied: astunparse~=1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (1.6.3)\n",
      "Requirement already satisfied: tensorflow-estimator<2.7,>=2.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (2.6.0)\n",
      "Requirement already satisfied: opt-einsum~=3.3.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (3.3.0)\n",
      "Requirement already satisfied: google-pasta~=0.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (0.2.0)\n",
      "Requirement already satisfied: h5py~=3.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (3.1.0)\n",
      "Requirement already satisfied: keras<2.7,>=2.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (2.6.0)\n",
      "Requirement already satisfied: clang~=5.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (5.0)\n",
      "Requirement already satisfied: keras-preprocessing~=1.1.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (1.1.2)\n",
      "Requirement already satisfied: joblib<0.15,>=0.12 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.5.0,>=1.4.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.14.1)\n",
      "Requirement already satisfied: pandas<2,>=1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.5.0,>=1.4.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.3.4)\n",
      "Requirement already satisfied: tensorflow-metadata<1.5,>=1.4 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.5.0,>=1.4.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.4.0)\n",
      "Requirement already satisfied: ipywidgets<8,>=7 in /opt/conda/lib/python3.7/site-packages (from tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (7.6.5)\n",
      "Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigtable<2,>=0.31.1->apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (0.12.3)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.1.2)\n",
      "Requirement already satisfied: cached-property in /opt/conda/lib/python3.7/site-packages (from h5py~=3.1.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<2.7,>=1.15.2->tfx==1.4.0->-r requirements.txt (line 1)) (1.5.2)\n",
      "Requirement already satisfied: docopt in /opt/conda/lib/python3.7/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (0.6.2)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (3.0.22)\n",
      "Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.1.3)\n",
      "Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (2.10.0)\n",
      "Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.7.5)\n",
      "Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (4.8.0)\n",
      "Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (5.1.1)\n",
      "Requirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.2.0)\n",
      "Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.18.0)\n",
      "Requirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (5.1.0)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (6.4.2)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.2.0)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.0.2)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (5.1.3)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (3.5.2)\n",
      "Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.7/site-packages (from oauth2client<5,>=2.0.1->apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (0.4.8)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (4.0.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.33->tfx==1.4.0->-r requirements.txt (line 1)) (2.10)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.6.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (1.8.0)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (2.0.2)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (3.3.4)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.4.6)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->jsonschema<4,>=3.0.1->kfp==1.8.1->-r requirements.txt (line 2)) (3.6.0)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx==1.4.0->-r requirements.txt (line 1)) (3.1.1)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.15.0)\n",
      "Requirement already satisfied: jupyter-client<8.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (7.0.6)\n",
      "Requirement already satisfied: argcomplete>=1.12.3 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.12.3)\n",
      "Requirement already satisfied: debugpy<2.0,>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.5.1)\n",
      "Requirement already satisfied: tornado<7.0,>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (6.1)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.16->ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.8.2)\n",
      "Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (4.9.1)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect>4.3->ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->keras-tuner<2,>=1.0.4->tfx==1.4.0->-r requirements.txt (line 1)) (0.2.5)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (6.4.5)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx==1.4.0->-r requirements.txt (line 1)) (2.20)\n",
      "Requirement already satisfied: entrypoints in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.3)\n",
      "Requirement already satisfied: nest-asyncio>=1.5 in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.5.1)\n",
      "Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (22.3.0)\n",
      "Requirement already satisfied: nbconvert in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (6.2.0)\n",
      "Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (21.1.0)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.12.1)\n",
      "Requirement already satisfied: Send2Trash>=1.5.0 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.8.0)\n",
      "Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.12.0)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (1.5.0)\n",
      "Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.1.2)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.8.4)\n",
      "Requirement already satisfied: bleach in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (4.1.0)\n",
      "Requirement already satisfied: testpath in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.5.0)\n",
      "Requirement already satisfied: defusedxml in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.7.1)\n",
      "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.5.4)\n",
      "Requirement already satisfied: webencodings in /opt/conda/lib/python3.7/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.36,>=0.35.0->tfx==1.4.0->-r requirements.txt (line 1)) (0.5.1)\n",
      "Building wheels for collected packages: kfp, docstring-parser, fire, kfp-server-api, strip-hints\n",
      "  Building wheel for kfp (setup.py): started\n",
      "  Building wheel for kfp (setup.py): finished with status 'done'\n",
      "  Created wheel for kfp: filename=kfp-1.8.1-py3-none-any.whl size=345340 sha256=2f7985b1afddd0806f7bdadb0355bf2993d7b0e646a491a2e3bf240b4d8e9367\n",
      "  Stored in directory: /root/.cache/pip/wheels/02/dd/71/45fa445fd9ea0c60ab0bd00b31760b1102ef2999f8002ee892\n",
      "  Building wheel for docstring-parser (pyproject.toml): started\n",
      "  Building wheel for docstring-parser (pyproject.toml): finished with status 'done'\n",
      "  Created wheel for docstring-parser: filename=docstring_parser-0.13-py3-none-any.whl size=31865 sha256=4aba5e28c4cb94be0e0fba69398ec4e61ff4e9c3e5c34b8d1bf6de37bace9cc1\n",
      "  Stored in directory: /root/.cache/pip/wheels/bd/88/3c/d1aa049309f7945178cac9fbe6561a86424f432da57c18ca0f\n",
      "  Building wheel for fire (setup.py): started\n",
      "  Building wheel for fire (setup.py): finished with status 'done'\n",
      "  Created wheel for fire: filename=fire-0.4.0-py2.py3-none-any.whl size=115943 sha256=8d1d9e00e28dbf74be38c27603a8a45622f7772a4c4c5d061ff71333ab8392de\n",
      "  Stored in directory: /root/.cache/pip/wheels/8a/67/fb/2e8a12fa16661b9d5af1f654bd199366799740a85c64981226\n",
      "  Building wheel for kfp-server-api (setup.py): started\n",
      "  Building wheel for kfp-server-api (setup.py): finished with status 'done'\n",
      "  Created wheel for kfp-server-api: filename=kfp_server_api-1.7.1-py3-none-any.whl size=92618 sha256=e9f814954bcc910b2f059efc21a5b7c7f6f246f5f81d2b4332fe604e15359554\n",
      "  Stored in directory: /root/.cache/pip/wheels/68/3f/d5/734c0278dd6c8969cef359edcf059505a61452c5eb0e2760e1\n",
      "  Building wheel for strip-hints (setup.py): started\n",
      "  Building wheel for strip-hints (setup.py): finished with status 'done'\n",
      "  Created wheel for strip-hints: filename=strip_hints-0.1.10-py2.py3-none-any.whl size=22302 sha256=4bd8a555357032e18e80a0d005a3590fa25302dada70b42756c7e54e7179f972\n",
      "  Stored in directory: /root/.cache/pip/wheels/5e/14/c3/6e44e9b2545f2d570b03f5b6d38c00b7534aa8abb376978363\n",
      "Successfully built kfp docstring-parser fire kfp-server-api strip-hints\n",
      "Installing collected packages: jsonschema, absl-py, tabulate, strip-hints, requests-toolbelt, kfp-server-api, google-cloud-aiplatform, fire, docstring-parser, Deprecated, cloudpickle, kfp\n",
      "  Attempting uninstall: jsonschema\n",
      "    Found existing installation: jsonschema 4.1.2\n",
      "    Uninstalling jsonschema-4.1.2:\n",
      "      Successfully uninstalled jsonschema-4.1.2\n",
      "  Attempting uninstall: absl-py\n",
      "    Found existing installation: absl-py 0.12.0\n",
      "    Uninstalling absl-py-0.12.0:\n",
      "      Successfully uninstalled absl-py-0.12.0\n",
      "  Attempting uninstall: google-cloud-aiplatform\n",
      "    Found existing installation: google-cloud-aiplatform 1.6.2\n",
      "    Uninstalling google-cloud-aiplatform-1.6.2:\n",
      "      Successfully uninstalled google-cloud-aiplatform-1.6.2\n",
      "  Attempting uninstall: cloudpickle\n",
      "    Found existing installation: cloudpickle 2.0.0\n",
      "    Uninstalling cloudpickle-2.0.0:\n",
      "      Successfully uninstalled cloudpickle-2.0.0\n",
      "\u001b[91mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tensorflow-io 0.18.0 requires tensorflow-io-gcs-filesystem==0.18.0, which is not installed.\n",
      "explainable-ai-sdk 1.3.2 requires xai-image-widget, which is not installed.\n",
      "tensorflow-io 0.18.0 requires tensorflow<2.6.0,>=2.5.0, but you have tensorflow 2.6.2 which is incompatible.\n",
      "WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n",
      "\u001b[0mSuccessfully installed Deprecated-1.2.13 absl-py-0.11.0 cloudpickle-1.6.0 docstring-parser-0.13 fire-0.4.0 google-cloud-aiplatform-1.7.1 jsonschema-3.2.0 kfp-1.8.1 kfp-server-api-1.7.1 requests-toolbelt-0.9.1 strip-hints-0.1.10 tabulate-0.8.9\n",
      "Removing intermediate container 3c0db03dcad5\n",
      " ---> 8b28648b69bf\n",
      "Step 4/5 : COPY tfx_taxifare_tips/ tfx_taxifare_tips/\n",
      " ---> a3640da07f5d\n",
      "Step 5/5 : ENV PYTHONPATH=\"/tfx_taxifare_tips:${PYTHONPATH}\"\n",
      " ---> Running in 2d17fd8fe1dc\n",
      "Removing intermediate container 2d17fd8fe1dc\n",
      " ---> 5b5ad17cd62c\n",
      "Successfully built 5b5ad17cd62c\n",
      "Successfully tagged us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips:latest\n",
      "PUSH\n",
      "Pushing us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips:latest\n",
      "The push refers to repository [us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips]\n",
      "ba90b5382c25: Preparing\n",
      "cc55b5367f15: Preparing\n",
      "a03f64bd036b: Preparing\n",
      "47eadc53848d: Preparing\n",
      "360afeadb83c: Preparing\n",
      "610c8c70c225: Preparing\n",
      "11aaa1ad9bbe: Preparing\n",
      "c1540b4fd089: Preparing\n",
      "5be20e773bf8: Preparing\n",
      "f0aed3b553c7: Preparing\n",
      "e55c134eccb1: Preparing\n",
      "b3e660ff020a: Preparing\n",
      "1795fd5db4de: Preparing\n",
      "dad41956ce77: Preparing\n",
      "ab61683fbd20: Preparing\n",
      "117941aa2d3f: Preparing\n",
      "3a4c764c9a45: Preparing\n",
      "34db85e87fb0: Preparing\n",
      "782857218c62: Preparing\n",
      "195f2f019955: Preparing\n",
      "9546e4c87db8: Preparing\n",
      "78613865172b: Preparing\n",
      "97676aba1403: Preparing\n",
      "0fb4d6eba0b6: Preparing\n",
      "e8b9f9e4195c: Preparing\n",
      "eb1f25078652: Preparing\n",
      "b7ce5de9980f: Preparing\n",
      "0796fbb63752: Preparing\n",
      "1233ef617acb: Preparing\n",
      "390ace10d575: Preparing\n",
      "c1b078f0e002: Preparing\n",
      "70720fcb790d: Preparing\n",
      "e05247f7d2f8: Preparing\n",
      "ab7c37becce8: Preparing\n",
      "5ccc1cf20429: Preparing\n",
      "b2d48dbbbef2: Preparing\n",
      "260b0b2ff58a: Preparing\n",
      "6babb56be259: Preparing\n",
      "610c8c70c225: Waiting\n",
      "11aaa1ad9bbe: Waiting\n",
      "c1b078f0e002: Waiting\n",
      "78613865172b: Waiting\n",
      "70720fcb790d: Waiting\n",
      "97676aba1403: Waiting\n",
      "e05247f7d2f8: Waiting\n",
      "0fb4d6eba0b6: Waiting\n",
      "e8b9f9e4195c: Waiting\n",
      "ab7c37becce8: Waiting\n",
      "eb1f25078652: Waiting\n",
      "b7ce5de9980f: Waiting\n",
      "5ccc1cf20429: Waiting\n",
      "0796fbb63752: Waiting\n",
      "c1540b4fd089: Waiting\n",
      "b2d48dbbbef2: Waiting\n",
      "1233ef617acb: Waiting\n",
      "260b0b2ff58a: Waiting\n",
      "390ace10d575: Waiting\n",
      "5be20e773bf8: Waiting\n",
      "6babb56be259: Waiting\n",
      "3a4c764c9a45: Waiting\n",
      "e55c134eccb1: Waiting\n",
      "b3e660ff020a: Waiting\n",
      "34db85e87fb0: Waiting\n",
      "ab61683fbd20: Waiting\n",
      "1795fd5db4de: Waiting\n",
      "117941aa2d3f: Waiting\n",
      "782857218c62: Waiting\n",
      "195f2f019955: Waiting\n",
      "dad41956ce77: Waiting\n",
      "9546e4c87db8: Waiting\n",
      "360afeadb83c: Layer already exists\n",
      "47eadc53848d: Layer already exists\n",
      "610c8c70c225: Layer already exists\n",
      "11aaa1ad9bbe: Layer already exists\n",
      "c1540b4fd089: Layer already exists\n",
      "5be20e773bf8: Layer already exists\n",
      "f0aed3b553c7: Layer already exists\n",
      "e55c134eccb1: Layer already exists\n",
      "b3e660ff020a: Layer already exists\n",
      "1795fd5db4de: Layer already exists\n",
      "dad41956ce77: Layer already exists\n",
      "ab61683fbd20: Layer already exists\n",
      "117941aa2d3f: Layer already exists\n",
      "a03f64bd036b: Pushed\n",
      "ba90b5382c25: Pushed\n",
      "34db85e87fb0: Layer already exists\n",
      "3a4c764c9a45: Layer already exists\n",
      "782857218c62: Layer already exists\n",
      "195f2f019955: Layer already exists\n",
      "9546e4c87db8: Layer already exists\n",
      "78613865172b: Layer already exists\n",
      "97676aba1403: Layer already exists\n",
      "0fb4d6eba0b6: Layer already exists\n",
      "e8b9f9e4195c: Layer already exists\n",
      "b7ce5de9980f: Layer already exists\n",
      "eb1f25078652: Layer already exists\n",
      "0796fbb63752: Layer already exists\n",
      "390ace10d575: Layer already exists\n",
      "1233ef617acb: Layer already exists\n",
      "c1b078f0e002: Layer already exists\n",
      "70720fcb790d: Layer already exists\n",
      "5ccc1cf20429: Layer already exists\n",
      "e05247f7d2f8: Layer already exists\n",
      "ab7c37becce8: Layer already exists\n",
      "b2d48dbbbef2: Layer already exists\n",
      "260b0b2ff58a: Layer already exists\n",
      "6babb56be259: Layer already exists\n",
      "cc55b5367f15: Pushed\n",
      "latest: digest: sha256:641d6ce8dcad2a89f1d6ac09af716d8a9a555e0b956052d9155c1e41a4a2bf45 size: 8306\n",
      "DONE\n",
      "--------------------------------------------------------------------------------\n",
      "ID                                    CREATE_TIME                DURATION  SOURCE                                                                                                IMAGES                                                                                           STATUS\n",
      "2b8a42f1-fa91-43ee-b1ac-3269318afa84  2021-12-10T04:57:43+00:00  4M55S     gs://dougkelly-vertex-demos_cloudbuild/source/1639112263.267746-dc49aada98f941ca8a3d522f7e57a263.tgz  us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips (+1 more)  SUCCESS\n"
     ]
    }
   ],
   "source": [
    "!gcloud builds submit --tag $TFX_IMAGE_URI . --timeout=20m --machine-type=e2-highcpu-8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36d61482-3a1c-40de-bf0d-1982637b6e58",
   "metadata": {},
   "source": [
    "### Compile the TFX pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "id": "4ba9bf02-ab96-4f35-b214-426115f558a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tfx_taxifare_tips"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "id": "8c55f083-c87b-4d13-87c9-d1dde8744f66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353')\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353?project=617979904441\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_PENDING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211210050353 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "# importlib.reload(tfx_taxifare_tips)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "id": "7ff0ce75-5af6-4154-ac9c-4ba70af01a3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "PIPELINE_DEFINITION_FILE = f'{config.PIPELINE_NAME}.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "id": "5c9ec875-4ef4-425f-bcd3-e1e7c5c38af6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:pipeline_root: gs://dougkelly-vertex-demos-tfx/tfx-artifacts/chicago-taxifare-tips-classifier-train-pipeline\n",
      "INFO:root:tfx_image_uri: us-central1-docker.pkg.dev/dougkelly-vertex-demos/tfx-taxifare-tips/tfx-taxifare-tips:latest\n",
      "INFO:absl:Excluding no splits because exclude_splits is not set.\n",
      "INFO:absl:Excluding no splits because exclude_splits is not set.\n",
      "INFO:root:Pipeline components: TrainDataGen, TestDataGen, SchemaImporter, StatisticsGen, ExampleValidator, Tranform, ModelTrainer, BaselineModelResolver, ModelEvaluator, ModelPusher\n",
      "INFO:root:Beam pipeline args: ['--project=dougkelly-vertex-demos', '--temp_location=gs://dougkelly-vertex-demos-tfx/temp', '--region=us-central1', '--runner=DataflowRunner']\n",
      "INFO:root:pipeline_info: PipelineInfo(pipeline_name: chicago-taxifare-tips-classifier-train-pipeline, pipeline_root: gs://dougkelly-vertex-demos-tfx/tfx-artifacts/chicago-taxifare-tips-classifier-train-pipeline, run_id: None)\n"
     ]
    }
   ],
   "source": [
    "from tfx_taxifare_tips.tfx_pipeline import pipeline_runner\n",
    "\n",
    "pipeline_definition = pipeline_runner.compile_training_pipeline(PIPELINE_DEFINITION_FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "id": "ac4b2f1f-3542-4f45-abad-fec15391aa8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n"
     ]
    }
   ],
   "source": [
    "pipeline_job = vertex_ai.pipeline_jobs.PipelineJob(\n",
    "          display_name=config.PIPELINE_NAME,\n",
    "          template_path=PIPELINE_DEFINITION_FILE,\n",
    "          pipeline_root=os.path.join(config.ARTIFACT_STORE_URI,config.PIPELINE_NAME)\n",
    "      )\n",
    "pipeline_job.run(sync=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e3f6dee-c9a2-4b42-9c27-a9b382be6c3f",
   "metadata": {},
   "source": [
    "### Extracting pipeline run metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "id": "b775b668-4567-473b-82cd-458248cf4b1c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>pipeline_name</th>\n",
       "      <td>chicago-taxifare-tips-classifier-train-pipeline</td>\n",
       "      <td>chicago-taxifare-tips-classifier-train-pipeline</td>\n",
       "      <td>chicago-taxifare-tips-classifier-train-pipeline</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run_name</th>\n",
       "      <td>chicago-taxifare-tips-classifier-train-pipelin...</td>\n",
       "      <td>chicago-taxifare-tips-classifier-train-pipelin...</td>\n",
       "      <td>chicago-taxifare-tips-classifier-train-pipelin...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                               0  \\\n",
       "pipeline_name    chicago-taxifare-tips-classifier-train-pipeline   \n",
       "run_name       chicago-taxifare-tips-classifier-train-pipelin...   \n",
       "\n",
       "                                                               1  \\\n",
       "pipeline_name    chicago-taxifare-tips-classifier-train-pipeline   \n",
       "run_name       chicago-taxifare-tips-classifier-train-pipelin...   \n",
       "\n",
       "                                                               2  \n",
       "pipeline_name    chicago-taxifare-tips-classifier-train-pipeline  \n",
       "run_name       chicago-taxifare-tips-classifier-train-pipelin...  "
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211208235005 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211208235005 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211208235005 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211208235005 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob projects/617979904441/locations/us-central1/pipelineJobs/chicago-taxifare-tips-classifier-train-pipeline-20211208235005 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n"
     ]
    }
   ],
   "source": [
    "pipeline_df = vertex_ai.get_pipeline_df(PIPELINE_NAME)\n",
    "pipeline_df = pipeline_df[pipeline_df.pipeline_name == PIPELINE_NAME]\n",
    "pipeline_df.T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb6946ca-303d-492a-985d-d6a3a83bd15b",
   "metadata": {},
   "source": [
    "### Upload trained model from Google Cloud Storage to Vertex AI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "id": "22bc4960-cca0-4439-99bc-8757d7e59c7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"Pipeline definition code.\"\"\"\n",
    "import os\n",
    "import sys\n",
    "import logging\n",
    "from typing import Text\n",
    "\n",
    "import tensorflow_model_analysis as tfma\n",
    "from tfx.proto import example_gen_pb2, transform_pb2, pusher_pb2\n",
    "from tfx.v1.types.standard_artifacts import Model, ModelBlessing, Schema\n",
    "from tfx.v1.extensions.google_cloud_big_query import BigQueryExampleGen\n",
    "from tfx.v1.extensions.google_cloud_ai_platform import Trainer as VertexTrainer\n",
    "from tfx.v1.dsl import Pipeline, Importer, Resolver, Channel\n",
    "from tfx.v1.dsl.experimental import LatestBlessedModelStrategy\n",
    "from tfx.v1.components import (\n",
    "    StatisticsGen,\n",
    "    ExampleValidator,\n",
    "    Transform,\n",
    "    Evaluator,\n",
    "    Pusher,\n",
    ")\n",
    "\n",
    "from tfx_taxifare_tips.tfx_pipeline import config\n",
    "from tfx_taxifare_tips.model_training import features, bq_datasource_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "bca058d0-2419-4d72-b6c7-b2fc45f0e047",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:absl:InteractiveContext metadata_connection_config not provided: using SQLite ML Metadata database at /home/jupyter/artifact-store/tfx-covertype-classifier/20211208_173153/metadata.sqlite.\n"
     ]
    }
   ],
   "source": [
    "import os, time\n",
    "from tfx.orchestration.experimental.interactive.interactive_context import (\n",
    "    InteractiveContext,\n",
    ")\n",
    "ARTIFACT_STORE = os.path.join(os.sep, \"home\", \"jupyter\", \"artifact-store\")\n",
    "SERVING_MODEL_DIR = os.path.join(os.sep, \"home\", \"jupyter\", \"serving_model\")\n",
    "DATA_ROOT = \"../../../data\"\n",
    "\n",
    "PIPELINE_NAME = \"tfx-covertype-classifier\"\n",
    "PIPELINE_ROOT = os.path.join(\n",
    "    ARTIFACT_STORE, PIPELINE_NAME, time.strftime(\"%Y%m%d_%H%M%S\")\n",
    ")\n",
    "\n",
    "os.makedirs(PIPELINE_ROOT, exist_ok=True)\n",
    "\n",
    "context = InteractiveContext(\n",
    "    pipeline_name=PIPELINE_NAME,\n",
    "    pipeline_root=PIPELINE_ROOT,\n",
    "    metadata_connection_config=None,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "id": "932b9f46-230d-47c5-a5c5-f24a0813937f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import_schema = Importer(\n",
    "    source_uri=\"tfx_taxifare_tips/raw_schema\",\n",
    "    artifact_type=Schema,\n",
    ").with_id(\"SchemaImporter\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "id": "e1bbcb79-f0ff-4573-8d01-be28621f1745",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:absl:Running driver for SchemaImporter\n",
      "INFO:absl:MetadataStore with DB connection initialized\n",
      "INFO:absl:Processing source uri: tfx_taxifare_tips/raw_schema, properties: {}, custom_properties: {}\n",
      "INFO:absl:Reusing existing artifact\n",
      "INFO:absl:Running executor for SchemaImporter\n",
      "INFO:absl:Running publisher for SchemaImporter\n",
      "INFO:absl:MetadataStore with DB connection initialized\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".tfx-object.expanded {\n",
       "  padding: 4px 8px 4px 8px;\n",
       "  background: white;\n",
       "  border: 1px solid #bbbbbb;\n",
       "  box-shadow: 4px 4px 2px rgba(0,0,0,0.05);\n",
       "}\n",
       ".tfx-object, .tfx-object * {\n",
       "  font-size: 11pt;\n",
       "}\n",
       ".tfx-object > .title {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".tfx-object .expansion-marker {\n",
       "  color: #999999;\n",
       "}\n",
       ".tfx-object.expanded > .title > .expansion-marker:before {\n",
       "  content: '▼';\n",
       "}\n",
       ".tfx-object.collapsed > .title > .expansion-marker:before {\n",
       "  content: '▶';\n",
       "}\n",
       ".tfx-object .class-name {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tfx-object .deemphasize {\n",
       "  opacity: 0.5;\n",
       "}\n",
       ".tfx-object.collapsed > table.attr-table {\n",
       "  display: none;\n",
       "}\n",
       ".tfx-object.expanded > table.attr-table {\n",
       "  display: block;\n",
       "}\n",
       ".tfx-object table.attr-table {\n",
       "  border: 2px solid white;\n",
       "  margin-top: 5px;\n",
       "}\n",
       ".tfx-object table.attr-table td.attr-name {\n",
       "  vertical-align: top;\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tfx-object table.attr-table td.attrvalue {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<script>\n",
       "function toggleTfxObject(element) {\n",
       "  var objElement = element.parentElement;\n",
       "  if (objElement.classList.contains('collapsed')) {\n",
       "    objElement.classList.remove('collapsed');\n",
       "    objElement.classList.add('expanded');\n",
       "  } else {\n",
       "    objElement.classList.add('collapsed');\n",
       "    objElement.classList.remove('expanded');\n",
       "  }\n",
       "}\n",
       "</script>\n",
       "<div class=\"tfx-object expanded\"><div class = \"title\" onclick=\"toggleTfxObject(this)\"><span class=\"expansion-marker\"></span><span class=\"class-name\">ExecutionResult</span><span class=\"deemphasize\"> at 0x7fe3a97a8f50</span></div><table class=\"attr-table\"><tr><td class=\"attr-name\">.execution_id</td><td class = \"attrvalue\">2</td></tr><tr><td class=\"attr-name\">.component</td><td class = \"attrvalue\">&lt;tfx.dsl.components.common.importer.Importer object at 0x7fe3a875eb90&gt;</td></tr><tr><td class=\"attr-name\">.component.inputs</td><td class = \"attrvalue\">{}</td></tr><tr><td class=\"attr-name\">.component.outputs</td><td class = \"attrvalue\"><table class=\"attr-table\"><tr><td class=\"attr-name\">['result']</td><td class = \"attrvalue\"><style>\n",
       ".tfx-object.expanded {\n",
       "  padding: 4px 8px 4px 8px;\n",
       "  background: white;\n",
       "  border: 1px solid #bbbbbb;\n",
       "  box-shadow: 4px 4px 2px rgba(0,0,0,0.05);\n",
       "}\n",
       ".tfx-object, .tfx-object * {\n",
       "  font-size: 11pt;\n",
       "}\n",
       ".tfx-object > .title {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".tfx-object .expansion-marker {\n",
       "  color: #999999;\n",
       "}\n",
       ".tfx-object.expanded > .title > .expansion-marker:before {\n",
       "  content: '▼';\n",
       "}\n",
       ".tfx-object.collapsed > .title > .expansion-marker:before {\n",
       "  content: '▶';\n",
       "}\n",
       ".tfx-object .class-name {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tfx-object .deemphasize {\n",
       "  opacity: 0.5;\n",
       "}\n",
       ".tfx-object.collapsed > table.attr-table {\n",
       "  display: none;\n",
       "}\n",
       ".tfx-object.expanded > table.attr-table {\n",
       "  display: block;\n",
       "}\n",
       ".tfx-object table.attr-table {\n",
       "  border: 2px solid white;\n",
       "  margin-top: 5px;\n",
       "}\n",
       ".tfx-object table.attr-table td.attr-name {\n",
       "  vertical-align: top;\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tfx-object table.attr-table td.attrvalue {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<script>\n",
       "function toggleTfxObject(element) {\n",
       "  var objElement = element.parentElement;\n",
       "  if (objElement.classList.contains('collapsed')) {\n",
       "    objElement.classList.remove('collapsed');\n",
       "    objElement.classList.add('expanded');\n",
       "  } else {\n",
       "    objElement.classList.add('collapsed');\n",
       "    objElement.classList.remove('expanded');\n",
       "  }\n",
       "}\n",
       "</script>\n",
       "<div class=\"tfx-object collapsed\"><div class = \"title\" onclick=\"toggleTfxObject(this)\"><span class=\"expansion-marker\"></span><span class=\"class-name\">Channel</span> of type <span class=\"class-name\">'Schema'</span> (1 artifact)<span class=\"deemphasize\"> at 0x7fe3a875eb10</span></div><table class=\"attr-table\"><tr><td class=\"attr-name\">.type_name</td><td class = \"attrvalue\">Schema</td></tr><tr><td class=\"attr-name\">._artifacts</td><td class = \"attrvalue\"><table class=\"attr-table\"><tr><td class=\"attr-name\">[0]</td><td class = \"attrvalue\"><style>\n",
       ".tfx-object.expanded {\n",
       "  padding: 4px 8px 4px 8px;\n",
       "  background: white;\n",
       "  border: 1px solid #bbbbbb;\n",
       "  box-shadow: 4px 4px 2px rgba(0,0,0,0.05);\n",
       "}\n",
       ".tfx-object, .tfx-object * {\n",
       "  font-size: 11pt;\n",
       "}\n",
       ".tfx-object > .title {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".tfx-object .expansion-marker {\n",
       "  color: #999999;\n",
       "}\n",
       ".tfx-object.expanded > .title > .expansion-marker:before {\n",
       "  content: '▼';\n",
       "}\n",
       ".tfx-object.collapsed > .title > .expansion-marker:before {\n",
       "  content: '▶';\n",
       "}\n",
       ".tfx-object .class-name {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tfx-object .deemphasize {\n",
       "  opacity: 0.5;\n",
       "}\n",
       ".tfx-object.collapsed > table.attr-table {\n",
       "  display: none;\n",
       "}\n",
       ".tfx-object.expanded > table.attr-table {\n",
       "  display: block;\n",
       "}\n",
       ".tfx-object table.attr-table {\n",
       "  border: 2px solid white;\n",
       "  margin-top: 5px;\n",
       "}\n",
       ".tfx-object table.attr-table td.attr-name {\n",
       "  vertical-align: top;\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tfx-object table.attr-table td.attrvalue {\n",
       "  text-align: left;\n",
       "}\n",
       "</style>\n",
       "<script>\n",
       "function toggleTfxObject(element) {\n",
       "  var objElement = element.parentElement;\n",
       "  if (objElement.classList.contains('collapsed')) {\n",
       "    objElement.classList.remove('collapsed');\n",
       "    objElement.classList.add('expanded');\n",
       "  } else {\n",
       "    objElement.classList.add('collapsed');\n",
       "    objElement.classList.remove('expanded');\n",
       "  }\n",
       "}\n",
       "</script>\n",
       "<div class=\"tfx-object collapsed\"><div class = \"title\" onclick=\"toggleTfxObject(this)\"><span class=\"expansion-marker\"></span><span class=\"class-name\">Artifact</span> of type <span class=\"class-name\">'Schema'</span> (uri: tfx_taxifare_tips/raw_schema)<span class=\"deemphasize\"> at 0x7fe3a9798890</span></div><table class=\"attr-table\"><tr><td class=\"attr-name\">.type</td><td class = \"attrvalue\">&lt;class &#x27;tfx.types.standard_artifacts.Schema&#x27;&gt;</td></tr><tr><td class=\"attr-name\">.uri</td><td class = \"attrvalue\">tfx_taxifare_tips/raw_schema</td></tr></table></div></td></tr></table></td></tr></table></div></td></tr></table></td></tr></table></div>"
      ],
      "text/plain": [
       "ExecutionResult(\n",
       "    component_id: SchemaImporter\n",
       "    execution_id: 2\n",
       "    outputs:\n",
       "        result: Channel(\n",
       "            type_name: Schema\n",
       "            artifacts: [Artifact(artifact: id: 1\n",
       "        type_id: 14\n",
       "        uri: \"tfx_taxifare_tips/raw_schema\"\n",
       "        custom_properties {\n",
       "          key: \"tfx_version\"\n",
       "          value {\n",
       "            string_value: \"1.4.0\"\n",
       "          }\n",
       "        }\n",
       "        create_time_since_epoch: 1638984854306\n",
       "        last_update_time_since_epoch: 1638984854306\n",
       "        , artifact_type: id: 14\n",
       "        name: \"Schema\"\n",
       "        )]\n",
       "            additional_properties: {}\n",
       "            additional_custom_properties: {}\n",
       "        ))"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "context.run(import_schema)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "id": "a6705ad7-f1dc-42a7-9c9f-a3712682af9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'tfx_taxifare_tips/raw_schema'"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import_schema.outputs[\"result\"].get()[0].uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec1f65cf-ab62-4c71-8ea2-52dc100ab811",
   "metadata": {},
   "outputs": [],
   "source": [
    "    examplevalidator = ExampleValidator(\n",
    "        statistics=statisticsgen.outputs[\"statistics\"],\n",
    "        schema=import_schema.outputs[\"result\"],\n",
    "    ).with_id(\"ExampleValidator\")"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m86",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m86"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}