courses/machine_learning/deepdive/09_sequence/labs/sinewaves.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Time Series Prediction\n",
    "\n",
    "**Objectives**\n",
    " 1. Build a linear, DNN and CNN model in Keras.\n",
    " 2. Build a simple RNN model and a multi-layer RNN model in Keras.\n",
    " \n",
    "In this lab we will start with a linear, DNN and CNN model \n",
    "\n",
    "Since the features of our model are sequential in nature, we'll next look at how to build various RNN models in Keras. We'll start with a simple RNN model and then see how to create a multi-layer RNN in Keras.\n",
    "\n",
    "We will be exploring a lot of different model types in this notebook.\n"
   ]
  },
  {
    "cell_type": "code",
    "metadata": {
      "id": "Nny3m465gKkY",
      "colab_type": "code",
      "colab": {}
    },
    "source": [
      "!sudo chown -R jupyter:jupyter /home/jupyter/training-data-analyst"
    ],
    "execution_count": null,
    "outputs": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
  "Collecting google-cloud-bigquery==1.25.0\n",
  "Downloading google_cloud_bigquery-1.25.0-py2.py3-none-any.whl (169 kB)\n",
     "|████████████████████████████████| 169 kB 4.7 MB/s eta 0:00:01\n",

  "Requirement already satisfied:  six in /home/jupyter/.local/lib/python3.7/site-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: google-auth in /usr/local/lib/python3.7/site-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: google-resumable-media in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: google-cloud-core in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: google-api-core in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: cachetools in /usr/local/lib/python3.7/dist-packages(from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: rsa in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: pyasn1-modules in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: googleapis-common-protos in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: pyasn1 in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from google-cloud-bigquery==1.25.0)\n",
  "Installing collected packages: google-resumable-media, google-cloud-bigquery\n",
  "\u001b[33mWARNING: You are using pip version 20.1; however, version 20.2.3 is available."
     ]
    }
   ],
   "source": [
    "!pip install --user google-cloud-bigquery==1.25.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: Restart your kernel to use updated packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kindly ignore the deprecation warnings and incompatibility errors related to google-cloud-storage."
   ]
  },
{
 "cell_type": "markdown",
 "metadata": {},
 "source": [
  "## Load necessary libraries and set up environment variables"
 ]
},
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT = \"your-gcp-project-here\" # REPLACE WITH YOUR PROJECT NAME\n",
    "BUCKET = \"your-gcp-bucket-here\" # REPLACE WITH YOUR BUCKET\n",
    "REGION = \"us-central1\" # REPLACE WITH YOUR BUCKET REGION e.g. us-central1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env \n",
    "PROJECT = PROJECT\n",
    "BUCKET = BUCKET\n",
    "REGION = REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "from google.cloud import bigquery\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import (Dense, DenseFeatures,\n",
    "                                     Conv1D, MaxPool1D,\n",
    "                                     Reshape, RNN,\n",
    "                                     LSTM, GRU, Bidirectional)\n",
    "from tensorflow.keras.callbacks import TensorBoard, ModelCheckpoint\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# For reproducible results.\n",
    "from numpy.random import seed\n",
    "seed(1)\n",
    "tf.random.set_seed(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore time series data\n",
    "\n",
    "We'll start by pulling a small sample of the time series data from Big Query and write some helper functions to clean up the data for modeling. We'll use the data from the `percent_change_sp500` table in BigQuery. The `close_values_prior_260` column contains the close values for any given stock for the previous 260 days. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 76 ms, sys: 20 ms, total: 96 ms\n",
      "Wall time: 1.59 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bq = bigquery.Client(project=PROJECT)\n",
    "\n",
    "bq_query = '''\n",
    "#standardSQL\n",
    "SELECT\n",
    "  symbol,\n",
    "  Date,\n",
    "  direction,\n",
    "  close_values_prior_260\n",
    "FROM\n",
    "  `stock_market.eps_percent_change_sp500`\n",
    "LIMIT\n",
    "  100\n",
    "'''\n"
  ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `clean_data` below does three things:\n",
    " 1. First, we'll remove any inf or NA values\n",
    " 2. Next, we parse the `Date` field to read it as a string.\n",
    " 3. Lastly, we convert the label `direction` into a numeric quantity, mapping 'DOWN' to 0, 'STAY' to 1 and 'UP' to 2. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_data(input_df):\n",
    "    \"\"\"Cleans data to prepare for training.\n",
    "\n",
    "    Args:\n",
    "        input_df: Pandas dataframe.\n",
    "    Returns:\n",
    "        Pandas dataframe.\n",
    "    \"\"\"\n",
    "    df = input_df.copy()\n",
    "\n",
    "    # Remove inf/na values.\n",
    "    real_valued_rows = ~(df == np.inf).max(axis=1)\n",
    "    df = df[real_valued_rows].dropna()\n",
    "\n",
    "    # TF doesn't accept datetimes in DataFrame.\n",
    "    df['Date'] = pd.to_datetime(df['Date'], errors='coerce')\n",
    "    df['Date'] = df['Date'].dt.strftime('%Y-%m-%d')\n",
    "\n",
    "    # TF requires numeric label.\n",
    "    df['direction_numeric'] = df['direction'].apply(lambda x: {'DOWN': 0,\n",
    "                                                               'STAY': 1,\n",
    "                                                               'UP': 2}[x])\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read data and preprocessing\n",
    "\n",
    "Before we begin modeling, we'll preprocess our features by scaling to the z-score. This will ensure that the range of the feature values being fed to the model are comparable and should help with convergence during gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "STOCK_HISTORY_COLUMN = 'close_values_prior_260'\n",
    "COL_NAMES = ['day_' + str(day) for day in range(0, 260)]\n",
    "LABEL = 'direction_numeric'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _scale_features(df):\n",
    "    \"\"\"z-scale feature columns of Pandas dataframe.\n",
    "\n",
    "    Args:\n",
    "        features: Pandas dataframe.\n",
    "    Returns:\n",
    "        Pandas dataframe with each column standardized according to the\n",
    "        values in that column.\n",
    "    \"\"\"\n",
    "    avg = df.mean()\n",
    "    std = df.std()\n",
    "    return (df - avg) / std\n",
    "\n",
    "\n",
    "def create_features(df, label_name):\n",
    "    \"\"\"Create modeling features and label from Pandas dataframe.\n",
    "\n",
    "    Args:\n",
    "        df: Pandas dataframe.\n",
    "        label_name: str, the column name of the label.\n",
    "    Returns:\n",
    "        Pandas dataframe\n",
    "    \"\"\"\n",
    "    # Expand 1 column containing a list of close prices to 260 columns.\n",
    "    time_series_features = df[STOCK_HISTORY_COLUMN].apply(pd.Series)\n",
    "\n",
    "    # Rename columns.\n",
    "    time_series_features.columns = COL_NAMES\n",
    "    time_series_features = _scale_features(time_series_features)\n",
    "\n",
    "    # Concat time series features with static features and label.\n",
    "    label_column = df[LABEL]\n",
    "\n",
    "    return pd.concat([time_series_features,\n",
    "                      label_column], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make train-eval-test split\n",
    "\n",
    "Next, we'll make repeatable splits for our train/validation/test datasets and save these datasets to local csv files. The query below will take a subsample of the entire dataset and then create a 70-15-15 split for the train/validation/test sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _create_split(phase):\n",
    "    \"\"\"Create string to produce train/valid/test splits for a SQL query.\n",
    "\n",
    "    Args:\n",
    "        phase: str, either TRAIN, VALID, or TEST.\n",
    "    Returns:\n",
    "        String.\n",
    "    \"\"\"\n",
    "    floor, ceiling = '2002-11-01', '2010-07-01'\n",
    "    if phase == 'VALID':\n",
    "        floor, ceiling = '2010-07-01', '2011-09-01'\n",
    "    elif phase == 'TEST':\n",
    "        floor, ceiling = '2011-09-01', '2012-11-30'\n",
    "    return '''\n",
    "    WHERE Date >= '{0}'\n",
    "    AND Date < '{1}'\n",
    "    '''.format(floor, ceiling)\n",
    "\n",
    "\n",
    "def create_query(phase):\n",
    "    \"\"\"Create SQL query to create train/valid/test splits on subsample.\n",
    "\n",
    "    Args:\n",
    "        phase: str, either TRAIN, VALID, or TEST.\n",
    "        sample_size: str, amount of data to take for subsample.\n",
    "    Returns:\n",
    "        String.\n",
    "    \"\"\"\n",
    "    basequery = \"\"\"\n",
    "    #standardSQL\n",
    "    SELECT\n",
    "      symbol,\n",
    "      Date,\n",
    "      direction,\n",
    "      close_values_prior_260\n",
    "    FROM\n",
    "      `stock_market.eps_percent_change_sp500`\n",
    "    \"\"\"\n",
    "    \n",
    "    return basequery + _create_split(phase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "For experimentation purposes, we'll train various models using data we can fit in memory using the `.csv` files we created above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_TIME_STEPS = 260\n",
    "N_LABELS = 3\n",
    "\n",
    "Xtrain = pd.read_csv('../stock-train.csv')\n",
    "Xvalid = pd.read_csv('../stock-valid.csv')\n",
    "\n",
    "ytrain = Xtrain.pop(LABEL)\n",
    "yvalid  = Xvalid.pop(LABEL)\n",
    "\n",
    "ytrain_categorical = to_categorical(ytrain.values)\n",
    "yvalid_categorical = to_categorical(yvalid.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To monitor training progress and compare evaluation metrics for different models, we'll use the function below to plot metrics captured from the training job such as training and validation loss or accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_curves(train_data, val_data, label='Accuracy'):\n",
    "    \"\"\"Plot training and validation metrics on single axis.\n",
    "\n",
    "    Args:\n",
    "        train_data: list, metrics obtrained from training data.\n",
    "        val_data: list, metrics obtained from validation data.\n",
    "        label: str, title and label for plot.\n",
    "    Returns:\n",
    "        Matplotlib plot.\n",
    "    \"\"\"\n",
    "    plt.plot(np.arange(len(train_data)) + 0.5,\n",
    "             train_data,\n",
    "             \"b.-\", label=\"Training \" + label)\n",
    "    plt.plot(np.arange(len(val_data)) + 1,\n",
    "             val_data, \"r.-\",\n",
    "             label=\"Validation \" + label)\n",
    "    plt.gca().xaxis.set_major_locator(mpl.ticker.MaxNLocator(integer=True))\n",
    "    plt.legend(fontsize=14)\n",
    "    plt.xlabel(\"Epochs\")\n",
    "    plt.ylabel(label)\n",
    "    plt.grid(True)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline\n",
    "\n",
    "Before we begin modeling in Keras, let's create a benchmark using a simple heuristic. Let's see what kind of accuracy we would get on the validation set if we predict the majority class of the training set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.29490392648287383"
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(yvalid == ytrain.value_counts().idxmax()) / yvalid.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. So just naively guessing the most common outcome `UP` will give about 29.5% accuracy on the validation set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear model\n",
    "\n",
    "We'll start with a simple linear model, mapping our sequential input to a single fully dense layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 241,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Dense(units=N_LABELS,\n",
    "                activation='softmax',\n",
    "                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))\n",
    "\n",
    "model.compile(optimizer=Adam(lr=0.001),\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(x=Xtrain.values,\n",
    "                    y=ytrain_categorical,\n",
    "                    batch_size=Xtrain.shape[0],\n",
    "                    validation_data=(Xvalid.values, yvalid_categorical),\n",
    "                    epochs=30,\n",
    "                    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['loss'],\n",
    "            history.history['val_loss'],\n",
    "            label='Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['accuracy'],\n",
    "            history.history['val_accuracy'],\n",
    "            label='Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy seems to level out pretty quickly. To report the accuracy, we'll average the accuracy on the validation set across the last few epochs of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3241512"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(history.history['val_accuracy'][-5:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deep Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The linear model is an improvement on our naive benchmark. Perhaps we can do better with a more complicated model. Next, we'll create a deep neural network with Keras. We'll experiment with a two layer DNN here but feel free to try a more complex model or add any other additional techniques to try an improve your performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [],
   "source": [
    "dnn_hidden_units = [16, 8]\n",
    "\n",
    "model = Sequential()\n",
    "for layer in dnn_hidden_units:\n",
    "    model.add(Dense(units=layer,\n",
    "                    activation=\"relu\"))\n",
    "\n",
    "model.add(Dense(units=N_LABELS,\n",
    "                activation=\"softmax\",\n",
    "                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))\n",
    "\n",
    "model.compile(optimizer=Adam(lr=0.001),\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(x=Xtrain.values,\n",
    "                    y=ytrain_categorical,\n",
    "                    batch_size=Xtrain.shape[0],\n",
    "                    validation_data=(Xvalid.values, yvalid_categorical),\n",
    "                    epochs=10,\n",
    "                    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['loss'],\n",
    "            history.history['val_loss'],\n",
    "            label='Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['accuracy'],\n",
    "            history.history['val_accuracy'],\n",
    "            label='Accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3669173"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(history.history['val_accuracy'][-5:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional Neural Network\n",
    "\n",
    "The DNN does slightly better. Let's see how a convolutional neural network performs. \n",
    "\n",
    "A 1-dimensional convolutional can be useful for extracting features from sequential data or deriving features from shorter, fixed-length segments of the data set. Check out the documentation for how to implement a [Conv1d in Tensorflow](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D). Max pooling is a downsampling strategy commonly used in conjunction with convolutional neural networks. Next, we'll build a CNN model in Keras using the `Conv1D` to create convolution layers and `MaxPool1D` to perform max pooling before passing to a fully connected dense layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "# Convolutional layer\n",
    "model.add(Reshape(target_shape=[N_TIME_STEPS, 1]))\n",
    "model.add(Conv1D(filters=5,\n",
    "                 kernel_size=5,\n",
    "                 strides=2,\n",
    "                 padding=\"valid\",\n",
    "                 input_shape=[None, 1]))\n",
    "model.add(MaxPool1D(pool_size=2,\n",
    "                    strides=None,\n",
    "                    padding='valid'))\n",
    "\n",
    "\n",
    "# Flatten the result and pass through DNN.\n",
    "model.add(tf.keras.layers.Flatten())\n",
    "model.add(Dense(units=N_TIME_STEPS//4,\n",
    "                activation=\"relu\"))\n",
    "\n",
    "model.add(Dense(units=N_LABELS, \n",
    "                activation=\"softmax\",\n",
    "                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))\n",
    "\n",
    "model.compile(optimizer=Adam(lr=0.01),\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(x=Xtrain.values,\n",
    "                    y=ytrain_categorical,\n",
    "                    batch_size=Xtrain.shape[0],\n",
    "                    validation_data=(Xvalid.values, yvalid_categorical),\n",
    "                    epochs=10,\n",
    "                    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['loss'],\n",
    "            history.history['val_loss'],\n",
    "            label='Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['accuracy'],\n",
    "            history.history['val_accuracy'],\n",
    "            label='Accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.34569758"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(history.history['val_accuracy'][-5:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recurrent Neural Network\n",
    "\n",
    "RNNs are particularly well-suited for learning sequential data. They retain state information from one iteration to the next by feeding the output from one cell as input for the next step. In the cell below, we'll build a RNN model in Keras. The final state of the RNN is captured and then passed through a fully connected layer to produce a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "# Reshape inputs to pass through RNN layer.\n",
    "model.add(Reshape(target_shape=[N_TIME_STEPS, 1]))\n",
    "model.add(LSTM(N_TIME_STEPS // 8,\n",
    "               activation='relu',\n",
    "               return_sequences=False))\n",
    "\n",
    "model.add(Dense(units=N_LABELS,\n",
    "                activation='softmax',\n",
    "                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))\n",
    "\n",
    "# Create the model.\n",
    "model.compile(optimizer=Adam(lr=0.001),\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(x=Xtrain.values,\n",
    "                    y=ytrain_categorical,\n",
    "                    batch_size=Xtrain.shape[0],\n",
    "                    validation_data=(Xvalid.values, yvalid_categorical),\n",
    "                    epochs=40,\n",
    "                    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['loss'],\n",
    "            history.history['val_loss'],\n",
    "            label='Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['accuracy'],\n",
    "            history.history['val_accuracy'],\n",
    "            label='Accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.35045946"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(history.history['val_accuracy'][-5:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-layer RNN\n",
    "\n",
    "Next, we'll build multi-layer RNN. Just as multiple layers of a deep neural network allow for more complicated features to be learned during training, additional RNN layers can potentially learn complex features in sequential data. For a multi-layer RNN the output of the first RNN layer is fed as the input into the next RNN layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_hidden_units = [N_TIME_STEPS // 16,\n",
    "                    N_TIME_STEPS // 32]\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "# Reshape inputs to pass through RNN layer.\n",
    "model.add(Reshape(target_shape=[N_TIME_STEPS, 1]))\n",
    "\n",
    "for layer in rnn_hidden_units[:-1]:\n",
    "    model.add(GRU(units=layer,\n",
    "                  activation='relu',\n",
    "                  return_sequences=True))\n",
    "\n",
    "model.add(GRU(units=rnn_hidden_units[-1],\n",
    "              return_sequences=False))\n",
    "model.add(Dense(units=N_LABELS,\n",
    "                activation=\"softmax\",\n",
    "                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))\n",
    "\n",
    "model.compile(optimizer=Adam(lr=0.001),\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "history = model.fit(x=Xtrain.values,\n",
    "                    y=ytrain_categorical,\n",
    "                    batch_size=Xtrain.shape[0],\n",
    "                    validation_data=(Xvalid.values, yvalid_categorical),\n",
    "                    epochs=50,\n",
    "                    verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['loss'],\n",
    "            history.history['val_loss'],\n",
    "            label='Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_curves(history.history['accuracy'],\n",
    "            history.history['val_accuracy'],\n",
    "            label='Accuracy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.36363825"
      ]
     },
     "execution_count": 221,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(history.history['val_accuracy'][-5:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2020 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}