courses/machine_learning/asl/05_review/labs/5_train.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training on Cloud AI Platform\n",
    "\n",
    "**Learning Objectives**\n",
    "- Use CAIP to run a distributed training job\n",
    "\n",
    "## Introduction \n",
    "After having testing our training pipeline both locally and in the cloud on a susbset of the data, we can submit another (much larger) training job to the cloud. It is also a good idea to run a hyperparameter tuning job to make sure we have optimized the hyperparameters of our model. \n",
    "\n",
    "This notebook illustrates how to do distributed training and hyperparameter tuning on Cloud AI Platform. \n",
    "\n",
    "To start, we'll set up our environment variables as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT = \"cloud-training-demos\"  # Replace with your PROJECT\n",
    "BUCKET = \"cloud-training-bucket\"  # Replace with your BUCKET\n",
    "REGION = \"us-central1\"            # Choose an available region for Cloud AI Platform\n",
    "TFVERSION = \"1.14\"                # TF version for CAIP to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"BUCKET\"] = BUCKET\n",
    "os.environ[\"PROJECT\"] = PROJECT\n",
    "os.environ[\"REGION\"] = REGION\n",
    "os.environ[\"TFVERSION\"] = TFVERSION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll look for the preprocessed data for the babyweight model and copy it over if it's not there. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "if ! gsutil ls -r gs://$BUCKET | grep -q gs://$BUCKET/babyweight/preproc; then\n",
    "    gsutil mb -l ${REGION} gs://${BUCKET}\n",
    "    # copy canonical set of preprocessed files if you didn't do previous notebook\n",
    "    gsutil -m cp -R gs://cloud-training-demos/babyweight gs://${BUCKET}\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gsutil ls gs://${BUCKET}/babyweight/preproc/*-00000*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous labs we developed our TensorFlow model and got it working on a subset of the data. Now we can package the TensorFlow code up as a Python module and train it on Cloud AI Platform.\n",
    "\n",
    "## Train on Cloud AI Platform\n",
    "\n",
    "Training on Cloud AI Platform requires two things:\n",
    "- Configuring our code as a Python package\n",
    "- Using gcloud to submit the training code to Cloud AI Platform\n",
    "\n",
    "### Move code into a Python package\n",
    "\n",
    "A Python package is simply a collection of one or more `.py` files along with an `__init__.py` file to identify the containing directory as a package. The `__init__.py` sometimes contains initialization code but for our purposes an empty file suffices.\n",
    "\n",
    "The bash command `touch` creates an empty file in the specified location, the directory `babyweight` should already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "touch babyweight/trainer/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then use the `%%writefile` magic to write the contents of the cell below to a file called `task.py` in the `babyweight/trainer` folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 1**\n",
    "\n",
    "The cell below write the file `babyweight/trainer/task.py` which sets up our training job. Here is where we determine which parameters of our model to pass as flags during training using the `parser` module. Look at how `batch_size` is passed to the model in the code below. Use this as an example to parse arguements for the following variables\n",
    "- `nnsize` which represents the hidden layer sizes to use for DNN feature columns\n",
    "- `nembeds` which represents the embedding size of a cross of n key real-valued parameters\n",
    "- `train_examples` which represents the number of examples (in thousands) to run the training job\n",
    "- `eval_steps` which represents the positive number of steps for which to evaluate model\n",
    "- `pattern` which specifies a pattern that has to be in input files. For example '00001-of' would process only one shard. For this variable, set 'of' to be the default. \n",
    "\n",
    "Be sure to include a default value for the parsed arguments above and specfy the `type` if necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile babyweight/trainer/task.py\n",
    "import argparse\n",
    "import json\n",
    "import os\n",
    "\n",
    "import tensorflow as tf\n",
    "\n",
    "from . import model\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\n",
    "        \"--bucket\",\n",
    "        help=\"GCS path to data. We assume that data is in \\\n",
    "        gs://BUCKET/babyweight/preproc/\",\n",
    "        required=True\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        \"--output_dir\",\n",
    "        help=\"GCS location to write checkpoints and export models\",\n",
    "        required=True\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        \"--batch_size\",\n",
    "        help=\"Number of examples to compute gradient over.\",\n",
    "        type=int,\n",
    "        default=512\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        \"--job-dir\",\n",
    "        help=\"this model ignores this field, but it is required by gcloud\",\n",
    "        default=\"junk\"\n",
    "    )\n",
    "    \n",
    "    # TODO: Your code goes here\n",
    "    \n",
    "    # TODO: Your code goes here\n",
    "    \n",
    "    # TODO: Your code goes here\n",
    "    \n",
    "    # TODO: Your code goes here\n",
    "    \n",
    "    # TODO: Your code goes here\n",
    "        \n",
    "    # Parse arguments\n",
    "    args = parser.parse_args()\n",
    "    arguments = args.__dict__\n",
    "\n",
    "    # Pop unnecessary args needed for gcloud\n",
    "    arguments.pop(\"job-dir\", None)\n",
    "\n",
    "    # Assign the arguments to the model variables\n",
    "    output_dir = arguments.pop(\"output_dir\")\n",
    "    model.BUCKET = arguments.pop(\"bucket\")\n",
    "    model.BATCH_SIZE = arguments.pop(\"batch_size\")\n",
    "    model.TRAIN_STEPS = (\n",
    "        arguments.pop(\"train_examples\") * 1000) / model.BATCH_SIZE\n",
    "    model.EVAL_STEPS = arguments.pop(\"eval_steps\")\n",
    "    print (\"Will train for {} steps using batch_size={}\".format(\n",
    "        model.TRAIN_STEPS, model.BATCH_SIZE))\n",
    "    model.PATTERN = arguments.pop(\"pattern\")\n",
    "    model.NEMBEDS = arguments.pop(\"nembeds\")\n",
    "    model.NNSIZE = arguments.pop(\"nnsize\")\n",
    "    print (\"Will use DNN size of {}\".format(model.NNSIZE))\n",
    "\n",
    "    # Append trial_id to path if we are doing hptuning\n",
    "    # This code can be removed if you are not using hyperparameter tuning\n",
    "    output_dir = os.path.join(\n",
    "        output_dir,\n",
    "        json.loads(\n",
    "            os.environ.get(\"TF_CONFIG\", \"{}\")\n",
    "        ).get(\"task\", {}).get(\"trial\", \"\")\n",
    "    )\n",
    "\n",
    "    # Run the training job\n",
    "    model.train_and_evaluate(output_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same way we can write to the file `model.py` the model that we developed in the previous notebooks. \n",
    "\n",
    "#### **Exercise 2**\n",
    "\n",
    "Complete the TODOs in the code cell below to create out `model.py`. We'll use the code we wrote for the Wide & Deep model. Look back at your `3_tensorflow_wide_deep` notebook and copy/paste the necessary code from that notebook into its place in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile babyweight/trainer/model.py\n",
    "import shutil\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.INFO)\n",
    "\n",
    "BUCKET = None  # set from task.py\n",
    "PATTERN = \"of\" # gets all files\n",
    "\n",
    "# Determine CSV and label columns\n",
    "# TODO: Your code goes here\n",
    "\n",
    "# Set default values for each CSV column\n",
    "# TODO: Your code goes here\n",
    "\n",
    "# Define some hyperparameters\n",
    "TRAIN_STEPS = 10000\n",
    "EVAL_STEPS = None\n",
    "BATCH_SIZE = 512\n",
    "NEMBEDS = 3\n",
    "NNSIZE = [64, 16, 4]\n",
    "\n",
    "# Create an input function reading a file using the Dataset API\n",
    "# Then provide the results to the Estimator API\n",
    "def read_dataset(prefix, mode, batch_size):\n",
    "    def _input_fn():\n",
    "        def decode_csv(value_column):\n",
    "            # TODO: Your code goes here\n",
    "        \n",
    "        # Use prefix to create file path\n",
    "        file_path = \"gs://{}/babyweight/preproc/{}*{}*\".format(\n",
    "            BUCKET, prefix, PATTERN)\n",
    "\n",
    "        # Create list of files that match pattern\n",
    "        file_list = tf.gfile.Glob(filename=file_path)\n",
    "\n",
    "        # Create dataset from file list\n",
    "        # TODO: Your code goes here\n",
    "        \n",
    "        # In training mode, shuffle the dataset and repeat indefinitely\n",
    "        # TODO: Your code goes here\n",
    " \n",
    "        dataset = # TODO: Your code goes here\n",
    "\n",
    "        # This will now return batches of features, label\n",
    "        return dataset\n",
    "    return _input_fn\n",
    "\n",
    "# Define feature columns\n",
    "def get_wide_deep():\n",
    "    # TODO: Your code goes here\n",
    "    return wide, deep\n",
    "\n",
    "\n",
    "# Create serving input function to be able to serve predictions later using provided inputs\n",
    "def serving_input_fn():\n",
    "    # TODO: Your code goes here\n",
    "    return tf.estimator.export.ServingInputReceiver(\n",
    "        features=features, receiver_tensors=feature_placeholders)\n",
    "\n",
    "# create metric for hyperparameter tuning\n",
    "def my_rmse(labels, predictions):\n",
    "    pred_values = predictions[\"predictions\"]\n",
    "    return {\"rmse\": tf.metrics.root_mean_squared_error(\n",
    "        labels=labels, predictions=pred_values)}\n",
    "\n",
    "# Create estimator to train and evaluate\n",
    "def train_and_evaluate(output_dir):\n",
    "    # TODO: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train locally\n",
    "\n",
    "After moving the code to a package, make sure it works as a standalone. Note, we incorporated the `--pattern` and `--train_examples` flags so that we don't try to train on the entire dataset while we are developing our pipeline. Once we are sure that everything is working on a subset, we can change the pattern so that we can train on all the data. Even for this subset, this takes about *3 minutes* in which you won't see any output ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 3**\n",
    "\n",
    "Fill in the missing code in the TODOs below so that we can run a very small training job over a single file (i.e. use the `pattern` equal to \"00000-of-\") with 1 train step and 1 eval step "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo \"bucket=${BUCKET}\"\n",
    "rm -rf babyweight_trained\n",
    "export PYTHONPATH=${PYTHONPATH}:${PWD}/babyweight\n",
    "python -m trainer.task \\\n",
    "    --bucket= # TODO: Your code goes here\n",
    "    --output_dir= # TODO: Your code goes here\n",
    "    --job-dir=./tmp \\\n",
    "    --pattern= # TODO: Your code goes here\n",
    "    --train_examples= # TODO: Your code goes here\n",
    "    --eval_steps= # TODO: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions\n",
    "\n",
    "The JSON below represents an input into your prediction model. Write the input.json file below with the next cell, then run the prediction locally to assess whether it produces predictions correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile inputs.json\n",
    "{\"is_male\": \"True\", \"mother_age\": 26.0, \"plurality\": \"Single(1)\", \"gestation_weeks\": 39}\n",
    "{\"is_male\": \"False\", \"mother_age\": 26.0, \"plurality\": \"Single(1)\", \"gestation_weeks\": 39}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 4**\n",
    "\n",
    "Finish the code in cell below to run a local prediction job on the `inputs.json` file we just created. You will need to provide two additional flags\n",
    "- one for `model-dir` specifying the location of the model binaries\n",
    "- one for `json-instances` specifying the location of the json file on which you want to predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "MODEL_LOCATION=$(ls -d $(pwd)/babyweight_trained/export/exporter/* | tail -1)\n",
    "echo $MODEL_LOCATION\n",
    "gcloud ai-platform local predict # TODO: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training on the Cloud with CAIP\n",
    "\n",
    "Once the code works in standalone mode, you can run it on Cloud AI Platform.  Because this is on the entire dataset, it will take a while. The training run took about <b> an hour </b> for me. You can monitor the job from the GCP console in the Cloud AI Platform section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 5**\n",
    "\n",
    "Look at the TODOs in the code cell below and fill in the missing information. Some of the required flags are already there for you. You will need to provide the rest. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "OUTDIR=gs://${BUCKET}/babyweight/trained_model\n",
    "JOBNAME=babyweight_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gsutil -m rm -rf $OUTDIR\n",
    "gcloud ai-platform jobs submit training $JOBNAME \\\n",
    "    --region= # TODO: Your code goes here\n",
    "    --module-name= # TODO: Your code goes here\n",
    "    --package-path= # TODO: Your code goes here\n",
    "    --job-dir= # TODO: Your code goes here\n",
    "    --staging-bucket=gs://$BUCKET \\\n",
    "    --scale-tier= #TODO: Your code goes here\n",
    "    --runtime-version= #TODO: Your code goes here\n",
    "    -- \\\n",
    "    --bucket=${BUCKET} \\\n",
    "    --output_dir=${OUTDIR} \\\n",
    "    --train_examples=200000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When I ran it, I used train_examples=2000000. When training finished, I filtered in the Stackdriver log on the word \"dict\" and saw that the last line was:\n",
    "<pre>\n",
    "Saving dict for global step 5714290: average_loss = 1.06473, global_step = 5714290, loss = 34882.4, rmse = 1.03186\n",
    "</pre>\n",
    "The final RMSE was 1.03 pounds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Optional: Hyperparameter tuning </h2>\n",
    "<p>\n",
    "All of these are command-line parameters to my program.  To do hyperparameter tuning, create hyperparam.xml and pass it as --configFile.\n",
    "This step will take <b>1 hour</b> -- you can increase maxParallelTrials or reduce maxTrials to get it done faster.  Since maxParallelTrials is the number of initial seeds to start searching from, you don't want it to be too large; otherwise, all you have is a random search.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 6**\n",
    "\n",
    "We need to create a .yaml file to pass with our hyperparameter tuning job. Fill in the TODOs below for each of the parameters we want to include in our hyperparameter search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%writefile hyperparam.yaml\n",
    "trainingInput:\n",
    "    scaleTier: STANDARD_1\n",
    "    hyperparameters:\n",
    "        hyperparameterMetricTag: rmse\n",
    "        goal: MINIMIZE\n",
    "        maxTrials: 20\n",
    "        maxParallelTrials: 5\n",
    "        enableTrialEarlyStopping: True\n",
    "        params:\n",
    "        - parameterName: batch_size\n",
    "          type: # TODO: Your code goes here\n",
    "          minValue: # TODO: Your code goes here\n",
    "          maxValue: # TODO: Your code goes here\n",
    "          scaleType: # TODO: Your code goes here\n",
    "        - parameterName: nembeds\n",
    "          type: # TODO: Your code goes here\n",
    "          minValue: # TODO: Your code goes here\n",
    "          maxValue: # TODO: Your code goes here\n",
    "          scaleType: # TODO: Your code goes here\n",
    "        - parameterName: nnsize\n",
    "          type: # TODO: Your code goes here\n",
    "          minValue: # TODO: Your code goes here\n",
    "          maxValue: # TODO: Your code goes here\n",
    "          scaleType: # TODO: Your code goes here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "OUTDIR=gs://${BUCKET}/babyweight/hyperparam\n",
    "JOBNAME=babyweight_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gsutil -m rm -rf $OUTDIR\n",
    "gcloud ai-platform jobs submit training $JOBNAME \\\n",
    "    --region=$REGION \\\n",
    "    --module-name=trainer.task \\\n",
    "    --package-path=$(pwd)/babyweight/trainer \\\n",
    "    --job-dir=$OUTDIR \\\n",
    "    --staging-bucket=gs://$BUCKET \\\n",
    "    --scale-tier=STANDARD_1 \\\n",
    "    --config=hyperparam.yaml \\\n",
    "    --runtime-version=$TFVERSION \\\n",
    "    -- \\\n",
    "    --bucket=${BUCKET} \\\n",
    "    --output_dir=${OUTDIR} \\\n",
    "    --eval_steps=10 \\\n",
    "    --train_examples=20000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Repeat training </h2>\n",
    "<p>\n",
    "This time with tuned parameters (note last line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "OUTDIR=gs://${BUCKET}/babyweight/trained_model_tuned\n",
    "JOBNAME=babyweight_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gsutil -m rm -rf $OUTDIR\n",
    "gcloud ai-platform jobs submit training $JOBNAME \\\n",
    "    --region=$REGION \\\n",
    "    --module-name=trainer.task \\\n",
    "    --package-path=$(pwd)/babyweight/trainer \\\n",
    "    --job-dir=$OUTDIR \\\n",
    "    --staging-bucket=gs://$BUCKET \\\n",
    "    --scale-tier=STANDARD_1 \\\n",
    "    --runtime-version=$TFVERSION \\\n",
    "    -- \\\n",
    "    --bucket=${BUCKET} \\\n",
    "    --output_dir=${OUTDIR} \\\n",
    "    --train_examples=20000 --batch_size=35 --nembeds=16 --nnsize=281"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2017 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}