notebooks/community/sdk/SDK_Tabular_Custom_Model_Training_asynchronous.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2020 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1c955a52adac"
      },
      "source": [
        "# Feedback or issues?\n",
        "For any feedback or questions, please open an [issue](https://github.com/googleapis/python-aiplatform/issues)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eHLV0D7Y5jtU"
      },
      "source": [
        "# Vertex SDK for Python: Custom Tabular Training (asynchronous) Example\n",
        "\n",
        "To use this Colaboratory notebook, you copy the notebook to your own Google Drive and open it with Colaboratory (or Colab). You can run each step, or cell, and see its results. To run a cell, use Shift+Enter. Colab automatically displays the return value of the last line in each cell. For more information about running notebooks in Colab, see the [Colab welcome page](https://colab.research.google.com/notebooks/welcome.ipynb).\n",
        "\n",
        "This notebook demonstrate how to create a custom model based on a tabular dataset (asynchronously). It will require you provide a bucket where the dataset CSV will be stored.\n",
        "\n",
        "Note: you may incur charges for training, prediction,  storage or usage of other GCP products in connection with testing this SDK."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lld3eeJUs5yM"
      },
      "source": [
        "# Install Vertex SDK for Python, Authenticate, and upload of a Dataset to your GCS bucket\n",
        "\n",
        "\n",
        "After the SDK installation the kernel will be automatically restarted. You may see this error message `Your session crashed for an unknown reason` which is normal."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sBfZtR4X1Dr_"
      },
      "outputs": [],
      "source": [
        "!pip3 uninstall -y google-cloud-aiplatform\n",
        "!pip3 install google-cloud-aiplatform\n",
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0SNmTBeD2nV"
      },
      "source": [
        "### Enter your project and GCS bucket\n",
        "\n",
        "Enter your Project Id in the cell below. Then run the cell to make sure the Cloud SDK uses the right project for all the commands in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f5be5dce7259"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kng9iKBwqcS5"
      },
      "outputs": [],
      "source": [
        "MY_PROJECT = \"YOUR PROJECT\"\n",
        "MY_STAGING_BUCKET = \"gs://YOUR BUCKET\"  # bucket should be in same region as Vertex AI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K-YsTXeQxXp7"
      },
      "source": [
        "The dataset we are using is the Abalone Dataset. For more information about this dataset please visit: https://archive.ics.uci.edu/ml/datasets/abalone"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V_T10yTTqcS_"
      },
      "outputs": [],
      "source": [
        "!wget https://storage.googleapis.com/download.tensorflow.org/data/abalone_train.csv\n",
        "!gsutil cp abalone_train.csv {MY_STAGING_BUCKET}/data/\n",
        "\n",
        "gcs_csv_path = f\"{MY_STAGING_BUCKET}/data/abalone_train.csv\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rk43VP_IqcTE"
      },
      "source": [
        "# Initialize Vertex SDK for Python\n",
        "\n",
        "Initialize the *client* for Vertex AI"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VCiC9gBWqcTF"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "\n",
        "aiplatform.init(project=MY_PROJECT, staging_bucket=MY_STAGING_BUCKET)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "35QVNhACqcTJ"
      },
      "source": [
        "# Create a Managed Tabular Dataset from CSV\n",
        "\n",
        "A Managed dataset can be used to create an AutoML model or a custom model. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4OfCqaYRqcTJ"
      },
      "outputs": [],
      "source": [
        "ds = aiplatform.TabularDataset.create(\n",
        "    display_name=\"abalone\", gcs_source=[gcs_csv_path], sync=False\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VcEOYYolqcTN"
      },
      "source": [
        "# Write Training Script\n",
        "- Write this cell as a file which will be used for custom training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OauJqJmJqcTO"
      },
      "outputs": [],
      "source": [
        "%%writefile training_script.py\n",
        "\n",
        "import pandas as pd\n",
        "import os\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "\n",
        "# uncomment and bump up replica_count for distributed training\n",
        "# strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()\n",
        "# tf.distribute.experimental_set_strategy(strategy)\n",
        "\n",
        "col_names = [\"Length\", \"Diameter\", \"Height\", \"Whole weight\", \"Shucked weight\", \"Viscera weight\", \"Shell weight\", \"Age\"]\n",
        "target = \"Age\"\n",
        "\n",
        "def aip_data_to_dataframe(wild_card_path):\n",
        "    return pd.concat([pd.read_csv(fp.numpy().decode(), names=col_names)\n",
        "                      for fp in tf.data.Dataset.list_files([wild_card_path])])\n",
        "\n",
        "def get_features_and_labels(df):\n",
        "    return df.drop(target, axis=1).values, df[target].values\n",
        "\n",
        "def data_prep(wild_card_path):\n",
        "    return get_features_and_labels(aip_data_to_dataframe(wild_card_path))\n",
        "\n",
        "\n",
        "model = tf.keras.Sequential([layers.Dense(64), layers.Dense(1)])\n",
        "model.compile(loss='mse', optimizer='adam')\n",
        "\n",
        "model.fit(*data_prep(os.environ[\"AIP_TRAINING_DATA_URI\"]),\n",
        "          epochs=10 ,\n",
        "          validation_data=data_prep(os.environ[\"AIP_VALIDATION_DATA_URI\"]))\n",
        "print(model.evaluate(*data_prep(os.environ[\"AIP_TEST_DATA_URI\"])))\n",
        "\n",
        "# save as Vertex AI Managed model\n",
        "tf.saved_model.save(model, os.environ[\"AIP_MODEL_DIR\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6-bBqipfqcTS"
      },
      "source": [
        "# Launch a Training Job to Create a Model\n",
        "\n",
        "Once we have defined your training script, we will create a model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "btb6d48lqcTT"
      },
      "outputs": [],
      "source": [
        "job = aiplatform.CustomTrainingJob(\n",
        "    display_name=\"train-abalone-dist-1-replica\",\n",
        "    script_path=\"training_script.py\",\n",
        "    container_uri=\"gcr.io/cloud-aiplatform/training/tf-cpu.2-2:latest\",\n",
        "    requirements=[\"gcsfs==0.7.1\"],\n",
        "    model_serving_container_image_uri=\"gcr.io/cloud-aiplatform/prediction/tf2-cpu.2-2:latest\",\n",
        ")\n",
        "model = job.run(ds, replica_count=1, model_display_name=\"abalone-model\", sync=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5vhDsMJNqcTW"
      },
      "source": [
        "# Deploy Your Model\n",
        "\n",
        "Deploy your model, then wait until the model FINISHES deployment before proceeding to prediction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y9GH72wWqcTX"
      },
      "outputs": [],
      "source": [
        "endpoint = model.deploy(machine_type=\"n1-standard-4\", sync=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nIw1ifPuqcTb"
      },
      "source": [
        "Wait for the deployment to complete"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b75WevFnWsNP"
      },
      "outputs": [],
      "source": [
        "endpoint.wait()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XIyKTohyPpSd"
      },
      "source": [
        "# Predict on the Endpoint"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3k6-rSZPqcTc"
      },
      "outputs": [],
      "source": [
        "prediction = endpoint.predict(\n",
        "    [\n",
        "        [0.435, 0.335, 0.11, 0.33399999999999996, 0.1355, 0.0775, 0.0965],\n",
        "        [0.585, 0.45, 0.125, 0.874, 0.3545, 0.2075, 0.225],\n",
        "    ]\n",
        ")\n",
        "prediction"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "AI_Platform_(Unified)_SDK_Tabular_Custom_Model_Training_(asynchronous).ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}