notebooks/sampling/monte_carlo/customer_price_index.ipynb (645 lines of code) (raw):
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "42dd4dd8",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"is_executing": true
}
},
"outputs": [],
"source": [
"# Copyright 2023 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"id": "aacec8ad",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"source": [
"# Customer Price Index simulation using Monte Carlo method"
]
},
{
"cell_type": "markdown",
"id": "4e57257c-3650-4d31-9d2a-e2dc7843246c",
"metadata": {},
"source": [
"<table align=\"left\">\n",
"\n",
"<a href=\"https://github.com/GoogleCloudPlatform/ai-ml-recipes/blob/main/notebooks/sampling/monte_carlo/customer_price_index.ipynb\">\n",
"<img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
"View on GitHub\n",
"</a>\n",
"</td>\n",
"<td>\n",
"<a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/ai-ml-recipes/main/notebooks/sampling/monte_carlo/customer_price_index.ipynb\">\n",
"<img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
"Open in Vertex AI Workbench\n",
"</a>\n",
"</td>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"id": "b98e384b",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"source": [
"### Overview"
]
},
{
"cell_type": "markdown",
"id": "2f746b57-73ba-4c41-8f70-28f3c4fe9d62",
"metadata": {},
"source": [
"#### Consumer Price Index (CPI)"
]
},
{
"cell_type": "markdown",
"id": "443d060d-efa8-4bec-adf4-55bd04ce3e8e",
"metadata": {},
"source": [
"The **Consumer Price Index (CPI)** is a measure of the average change in prices paid by consumers for a basket of goods and services. **Inflation** is the rate at which the general level of prices for goods and services is rising and, consequently, the purchasing power of currency is falling.\n",
"\n",
"The **CPI** is a key measure of inflation because it tracks the prices of a wide range of goods and services that are purchased by consumers. If the CPI increases, it means that the prices of goods and services have increased on average. This is a sign of inflation.\n",
"\n",
"For example, if the **CPI increases by 2% in a year**, it means that the **average price of goods and services has increased by 2% over that year**. This means that consumers will need to spend 2% more money to buy the same basket of goods and services that they bought the previous year.\n",
"\n",
"The CPI is used by a variety of economic actors, including governments, businesses, and consumers. Governments use the CPI to set economic policy, such as interest rates. Businesses use the CPI to make pricing decisions. Consumers use the CPI to track the cost of living."
]
},
{
"cell_type": "markdown",
"id": "92f142bd-3fc9-46b4-a192-668e2e1edd3d",
"metadata": {},
"source": [
"#### Monte Carlo method"
]
},
{
"cell_type": "markdown",
"id": "a8ff2c06-fc9d-4d10-93d0-b64efefe041c",
"metadata": {},
"source": [
"This code uses the **Monte Carlo** method to simulate the future path of the **Consumer Price Index (CPI)**. The code begins by calculating the historical mean and standard deviation of the percentage change in the CPI. These values are then used to generate a large number of **simulated paths for the future CPI**. Each simulated path is generated by starting with the **current value of the CPI and then adding a series of normally distributed random increments**. The size of the random increments is determined by the historical **mean and standard deviation of the percentage change in the CPI**.\n",
"\n",
"Once the simulated paths have been generated, they are used to calculate the **probability of various future events**. For example, the code can be used to calculate the **probability that the inflation rate in a given year will be below a certain target**. The **Monte Carlo** method is a **powerful tool for forecasting** future economic conditions. By simulating a large number of possible future paths, the Monte Carlo method can provide a more accurate picture of the range of possible outcomes than traditional forecasting methods.\n",
"\n",
"The concepts and theory related to the Monte Carlo method are based on the theory of **probability and statistics**. The Monte Carlo method works by **randomly sampling from a probability distribution**. In the case of the CPI, the probability distribution is the distribution of the percentage change in the CPI. By randomly sampling from this distribution, the Monte Carlo method can generate a large number of possible future paths for the CPI.\n",
"\n",
"The Monte Carlo method is a versatile tool that can be used to forecast a wide range of economic variables. In addition to forecasting the CPI, the Monte Carlo method can be used to forecast interest rates, exchange rates, and stock prices. The Monte Carlo method is a valuable tool for anyone who is interested in forecasting future economic conditions.\n",
"\n",
"- This code utilizes the **Monte Carlo** simulation technique to model the future path of the **Consumer Price Index (CPI)** and estimate the probability of inflation falling below a specific threshold.\n",
"- It employs the **geometric Brownian motion model**, a stochastic process commonly used to represent financial asset prices, to simulate numerous potential trajectories of the CPI.\n",
"- The code extracts historical CPI data, calculates essential parameters like mean and standard deviation, and generates random Wiener increments to simulate the unpredictable nature of market movements.\n",
"- It then constructs a DataFrame of simulated CPI paths and visualizes the 80th, 50th, and 20th percentiles of these paths.\n",
"- Finally, it estimates the probability of inflation falling below 2% in a specific year by analyzing the simulated inflation rates."
]
},
{
"cell_type": "markdown",
"id": "ab629034-3a72-4f56-b259-3e83730fe04a",
"metadata": {},
"source": [
"#### Pandas API on Spark"
]
},
{
"cell_type": "markdown",
"id": "7d4f73d8-8cbc-46ae-a939-0744c598f03c",
"metadata": {},
"source": [
"\n",
"The code is using the Pandas API on Spark. This is a feature of PySpark that allows users to write code using the familiar Pandas DataFrame API on Spark DataFrames. This can be a significant advantage for users who are already familiar with Pandas, as it allows them to use their existing knowledge to work with Spark DataFrames.\n",
"\n",
"The Pandas API on Spark is implemented by converting Spark DataFrames to Pandas DataFrames under the hood. This means that all of the Pandas DataFrame methods are available for use on Spark DataFrames. However, it is important to note that not all Pandas methods are supported by the Pandas API on Spark. A list of supported methods can be found in the PySpark documentation.\n",
"\n",
"The Pandas API on Spark can be a useful tool for users who want to take advantage of the power of Spark while still using the familiar Pandas API. This can be especially useful for users who are already familiar with Pandas and who do not want to learn the Spark DataFrame API.\n",
"\n",
"In the specific case of the code being analyzed, the Pandas API on Spark is being used to generate a large number of simulated paths for the future CPI. This is done by using the pandas.DataFrame.pct_change() method to calculate the percentage change in the CPI. The percentage change is then used to generate a series of normally distributed random increments. These random increments are then added to the current value of the CPI to generate a simulated path for the future CPI."
]
},
{
"cell_type": "markdown",
"id": "b4307bfa-0ff9-4ded-a2a8-5754dfb67122",
"metadata": {},
"source": [
"### Setup"
]
},
{
"cell_type": "markdown",
"id": "5b6688a6-af15-4747-9387-60d38cdf410a",
"metadata": {},
"source": [
"Since we are using the [Pandas API on Spark](https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html) make sure your runtime uses Spark version >= 3.2.0"
]
},
{
"cell_type": "markdown",
"id": "5550ff0b-0e44-42bb-8159-77685d955610",
"metadata": {},
"source": [
"#### Identity and Access Management (IAM)"
]
},
{
"cell_type": "markdown",
"id": "68a9fb5c-0e29-48d7-aced-a98b0b6aef80",
"metadata": {},
"source": [
"Make sure the service account running this notebook has the required permissions:\n",
"\n",
"- **Run the notebook**\n",
" - AI Platform Notebooks Service Agent\n",
" - Notebooks Admin\n",
" - Vertex AI Administrator\n",
"- **Read files from bucket**\n",
" - Storage Object Viewer\n",
"- **Run Dataproc jobs**\n",
" - Dataproc Service Agent\n",
" - Dataproc Worker"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9d36432b",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"is_executing": true
}
},
"outputs": [],
"source": [
"#### Import dependencies\n",
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"from pyspark.sql.functions import col\n",
"\n",
"PYARROW_IGNORE_TIMEZONE = 1;"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7de87682",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"from pyspark.sql import SparkSession"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e998745",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
},
"pycharm": {
"is_executing": true
}
},
"outputs": [],
"source": [
"spark = SparkSession.builder \\\n",
" .appName(\"Customer Price Index Monte Carlo Simulation\") \\\n",
" .enableHiveSupport() \\\n",
" .getOrCreate()\n",
"\n",
"spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'true')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "11bad300",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"outputs": [],
"source": [
"raw_dataset = spark.read.option(\"header\",True).csv(\"gs://dataproc-metastore-public-binaries/us_customer_price_index_yearly/\")"
]
},
{
"cell_type": "markdown",
"id": "071ac7ec",
"metadata": {
"collapsed": false,
"jupyter": {
"outputs_hidden": false
}
},
"source": [
"### Pre-process dataset / filter values"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "18af9feb-40fa-4fdb-b0a7-fcdf9e1888a1",
"metadata": {},
"outputs": [],
"source": [
"cpi_80 = raw_dataset[raw_dataset.Year >= 1980]\n",
"cpi_80 = cpi_80.pandas_api(index_col = 'Year')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d2f4c359-ed8e-41ec-adfa-2be110d6ff05",
"metadata": {},
"outputs": [],
"source": [
"inf = cpi_80.pct_change().dropna()\n",
"inf.head(10)"
]
},
{
"cell_type": "markdown",
"id": "660f2b82-f871-4880-bcb5-a9b60b503f93",
"metadata": {},
"source": [
"### Run Monte Carlos simulation"
]
},
{
"cell_type": "markdown",
"id": "d26a1723-bd1f-4e9c-8b7a-13fd762d28f0",
"metadata": {},
"source": [
"**Formula:** CPI_t = CPI_0 * exp((μ - 0.5 * σ^2) * t + σ * W_t)\n",
"\n",
"where:\n",
"\n",
"- CPI_t is the CPI at time t\n",
"- CPI_0 is the CPI at time 0\n",
"- μ is the drift rate of the CPI\n",
"- σ is the volatility of the CPI\n",
"- W_t is a Wiener process\n",
"\n",
"This formula is used to model the CPI under geometric Brownian motion. \n",
"Geometric Brownian motion is a stochastic process that is often used to model the prices of financial assets and other economic variables. \n",
"\n",
"The Monte Carlo method works by generating a large number of random samples from the probability distribution of the Wiener process. \n",
"These random samples are then used to generate a large number of simulated paths for the future CPI. \n",
"The simulated paths are then used to estimate the probability of various future events, such as the probability that the inflation rate in a given year will be below a certain target. \n",
"\n",
"In this case, the price is the CPI. The Monte Carlo method is being used to generate a large number of simulated paths for the future CPI. The simulated paths are then used to estimate the probability of various future events, such as the probability that the inflation rate in a given year will be below a certain target."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ea313c97-2998-4f0a-9c8e-7ef33c98c3be",
"metadata": {},
"outputs": [],
"source": [
"n = 10\n",
"t = 10\n",
"number_simulations = 10**6"
]
},
{
"cell_type": "markdown",
"id": "81d8acd7-8e06-4b5d-9414-9dc9f51f2da9",
"metadata": {},
"source": [
"- **n** is assigned the value 10, representing the number of time steps. This is the number of discrete points in time that will be used to model the future path of the CPI.\n",
"- **t** is assigned the value 10, representing the total time period. This is the length of time over which the future path of the CPI will be modeled.\n",
"- **number_simulations** is assigned the value 10**6, representing the number of Monte Carlo simulations to be performed. This is the number of times that the future path of the CPI will be simulated."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4cc5fdf4-6458-4e93-ad57-00f39797c2f7",
"metadata": {},
"outputs": [],
"source": [
"mu = inf['CPI'].mean()\n",
"sigma = inf['CPI'].std()"
]
},
{
"cell_type": "markdown",
"id": "66f3c30f-41a9-401a-81af-0bc33028377f",
"metadata": {},
"source": [
"- **mu** is assigned the mean of the CPI column in the inf DataFrame. This is the historical mean of the percentage change in the CPI.\n",
"- **sigma** is assigned the standard deviation of the CPI column in the inf DataFrame. This is the historical standard deviation of the percentage change in the CPI."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d155e817-f1f3-4bc9-b387-df3e221ace73",
"metadata": {},
"outputs": [],
"source": [
"cpi_index = cpi_80.index.tolist()\n",
"v0 = cpi_80.CPI[cpi_index[-1]]"
]
},
{
"cell_type": "markdown",
"id": "ca8d689d-8d40-4156-8670-6a5d32192d38",
"metadata": {},
"source": [
"- **cpi_index** is assigned a list of the indices in the cpi_80 DataFrame.\n",
"- **v0** is assigned the value of the CPI column in the cpi_80 DataFrame at the last index in cpi_index. This is the current value of the CPI."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ba5764a4-4459-461d-beb3-3e6faf4d8d95",
"metadata": {},
"outputs": [],
"source": [
"dt = t/n\n",
"dw_ant = np.random.normal(scale=np.sqrt(dt), size=(int(number_simulations/2), n))\n",
"dw = np.concatenate((dw_ant, -dw_ant), axis=0)\n",
"dw = np.random.normal(scale=np.sqrt(dt), size=(number_simulations, n))"
]
},
{
"cell_type": "markdown",
"id": "dbb9601e-b82f-4112-8ea5-3a708222e726",
"metadata": {},
"source": [
"- **dt** is assigned the value of **t** divided by **n**, representing the time step size.\n",
"- **dw_ant** is assigned a NumPy array of normally distributed random numbers with a scale of **np.sqrt(dt)** and a size of **(int(number_simulations/2), n)**. These are the Wiener increments for the first half of the Monte Carlo simulations.\n",
"- **dw** is assigned the concatenation of **dw_ant** and its negative, along the axis 0. This ensures that the Wiener increments are antithetic, meaning that the sum of the Wiener increments over any time interval is zero. \n",
"- **dw** is reassigned a NumPy array of normally distributed random numbers with a scale of **np.sqrt(dt)** and a size of **(number_simulations, n)**. These are the Wiener increments for all of the Monte Carlo simulations."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "debc8972-f098-4247-91e6-3ba03f574a7a",
"metadata": {},
"outputs": [],
"source": [
"w = np.cumsum(dw, axis=1)"
]
},
{
"cell_type": "markdown",
"id": "2e54aa22-960e-4520-86fe-fe4f30a4d23f",
"metadata": {},
"source": [
"- **w** is assigned the cumulative sum of **dw** along the axis 1. This is the Wiener process, which is a continuous-time stochastic process that is used to model Brownian motion."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ec4155ec-c750-496b-963f-8083d0479ac9",
"metadata": {},
"outputs": [],
"source": [
"time_step = np.linspace(dt, t, n)\n",
"time_steps = np.broadcast_to(time_step, (number_simulations, n))"
]
},
{
"cell_type": "markdown",
"id": "9aa333c6-ea6a-4448-8750-d2ba999ee423",
"metadata": {},
"source": [
"- **time_step** is assigned a NumPy array of linearly spaced numbers between **dt** and **t**, with **n** elements. This is the vector of time steps.\n",
"- **time_steps** is assigned the result of broadcasting time_step to a shape of **(number_simulations, n)**. This ensures that each Monte Carlo simulation has its own vector of time steps."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f7d25d5a-3a39-4512-9876-23f1c428e706",
"metadata": {},
"outputs": [],
"source": [
"vt = v0 * np.exp((mu - 0.5 * sigma ** 2) * time_steps + sigma * w)"
]
},
{
"cell_type": "markdown",
"id": "e865648f-83ee-48f8-bb70-ad2ac02c3398",
"metadata": {},
"source": [
"- **vt** is assigned the value of v0 multiplied by the exponential of **(mu - 0.5 * sigma ** 2) * time_steps + sigma * w**. This is the solution to the stochastic differential equation for the geometric Brownian motion model."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "73807f96-65be-4871-8870-49d2e82be45e",
"metadata": {},
"outputs": [],
"source": [
"vt = np.insert(vt, 0, v0, axis=1)"
]
},
{
"cell_type": "markdown",
"id": "8678fb66-b9aa-4a51-8e9d-856b3957871b",
"metadata": {},
"source": [
"- **vt** is reassigned the result of inserting v0 into the first column of vt. This ensures that the first value in each simulated path is the current value of the CPI."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f1e2995c-970d-43c5-9e77-76d7a2788395",
"metadata": {},
"outputs": [],
"source": [
"cpi_index = cpi_80.index.tolist()\n",
"index = range(cpi_index[-1], cpi_index[-1] + n + 1)"
]
},
{
"cell_type": "markdown",
"id": "eb1d6c5b-b576-4c34-84ea-067f01dc430f",
"metadata": {},
"source": [
"- **index** is assigned a range of integers from the last index in cpi_index to the last index in cpi_index plus n plus 1. This is the index for the simulated paths."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "9ce0684e-d0d0-4892-b813-9b23df30cc89",
"metadata": {},
"outputs": [],
"source": [
"paths = pd.DataFrame(np.transpose(vt), index=index)\n",
"mean = paths.mean(axis=1)"
]
},
{
"cell_type": "markdown",
"id": "8e4e97f1-1dc7-4086-90f3-448d02f40467",
"metadata": {},
"source": [
"- **paths** is assigned a Pandas DataFrame with the transposed values of vt as the data and index as the index. This is the DataFrame of simulated paths for the CPI.\n",
"- **mean** is assigned the mean of paths along the axis 1. This is the mean simulated path for the CPI."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7ca4724f-4de7-4e16-b7e8-a4ca00ed2156",
"metadata": {},
"outputs": [],
"source": [
"paths.head(11)"
]
},
{
"cell_type": "markdown",
"id": "3ffe3c66-45a3-4126-bc0f-f985569d0cb2",
"metadata": {},
"source": [
"The following plot shows the range of possible future paths for the CPI. \n",
"- The 80th percentile represents the value above which 80% of the simulated paths fall.\n",
"- The 50th percentile represents the median value of the simulated paths.\n",
"- The 20th percentile represents the value below which 20% of the simulated paths fall.\n",
"\n",
"The red line in the plot shows the mean of the simulated paths. The mean is the average value of all of the simulated paths. \n",
"The green and blue lines in the plot show the 20th and 80th percentiles of the simulated paths. \n",
"\n",
"The area between the green and blue lines represents the range of values within which 60% of the simulated paths fall. \n",
"\n",
"This plot can be used to visualize the range of possible future paths for the CPI. \n",
"The plot can also be used to estimate the probability of various future events. \n",
"For example, the probability that the CPI will be above a certain value in a given year can be estimated by looking at the percentage of simulated paths that are above that value. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5df10b09-b674-4194-b1a1-0c21c11b8de6",
"metadata": {},
"outputs": [],
"source": [
"pd.options.display.float_format = '{:,.2f}'.format\n",
"paths.quantile(0.8, axis=1).plot(figsize=(20, 8))\n",
"paths.mean(axis=1).plot(color='red')\n",
"paths.quantile(0.2, axis=1).plot(color='green')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0cceb43e-5e98-44a4-8c4a-2937f9ad8131",
"metadata": {},
"outputs": [],
"source": [
"predicted_inf = paths.pct_change().dropna()\n",
"predicted_inf.head(11)"
]
},
{
"cell_type": "markdown",
"id": "ebad0e02-347c-4ecb-9499-6b95ea1827b6",
"metadata": {},
"source": [
"- **predicted_inf** is assigned the percentage change of the paths DataFrame, with any missing values dropped. This is the DataFrame of simulated paths for the inflation rate."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3b58657e-faa0-42f6-b07f-e39cf6cafdb8",
"metadata": {},
"outputs": [],
"source": [
"inf_2025 = predicted_inf.loc[2025, :].values\n",
"print(\"Estimated probability that the inflation rate in 2025 will be less than 2% in percetage: \", np.count_nonzero(inf_2025 < 0.02) / 1e6 * 100)"
]
},
{
"cell_type": "markdown",
"id": "a4323280-f74a-4ffa-849e-b22bb2cac12d",
"metadata": {},
"source": [
"**inf_2025** is assigned the values of the predicted_inf DataFrame at the index 2025. This is the vector of simulated inflation rates for the year 2025. \n",
"The number of simulated inflation rates for the year 2025 that are less than 2% is counted. \n",
"The count is divided by the total number of simulated inflation rates and multiplied by 100 to express the result as a percentage. \n",
"This is the estimated probability that the inflation rate in 2025 will be less than 2%. "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 on cluster-dpms (Remote)",
"language": "python",
"name": "7a42ae6b283a613cfabce6f7-python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}