training/sagemaker-debugger/callback_bottleneck.ipynb (832 lines of code) (raw):
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -qU horovod"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.011952,
"end_time": "2021-06-01T00:13:00.123603",
"exception": false,
"start_time": "2021-06-01T00:13:00.111651",
"status": "completed"
},
"tags": []
},
"source": [
"# Identify a CPU bottleneck caused by a callback process with Amazon SageMaker Debugger \n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
"\n",
"\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.011952,
"end_time": "2021-06-01T00:13:00.123603",
"exception": false,
"start_time": "2021-06-01T00:13:00.111651",
"status": "completed"
},
"tags": []
},
"source": [
"\n",
"In this notebook we demonstrate how to identify a training bottleneck that is caused by a TensorFlow Keras callback.\n",
"To simulate this type of bottleneck, we will program the callback associated with the tensor monitoring feature of Amazon SageMaker Debugger, to collect an excessive number of tensors, and at a high frequency."
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.011776,
"end_time": "2021-06-01T00:13:00.147143",
"exception": false,
"start_time": "2021-06-01T00:13:00.135367",
"status": "completed"
},
"tags": []
},
"source": [
"### Install sagemaker\n",
"To use the new Debugger profiling features, ensure that you have the latest version of SageMaker SDK installed. The following cell updates the library and restarts the Jupyter kernel to apply the updates."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:00.175312Z",
"iopub.status.busy": "2021-06-01T00:13:00.174799Z",
"iopub.status.idle": "2021-06-01T00:13:00.177135Z",
"shell.execute_reply": "2021-06-01T00:13:00.176718Z"
},
"papermill": {
"duration": 0.018328,
"end_time": "2021-06-01T00:13:00.177237",
"exception": false,
"start_time": "2021-06-01T00:13:00.158909",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"import sys\n",
"import IPython\n",
"install_needed = False # should only be True once\n",
"if install_needed:\n",
" print(\"installing deps and restarting kernel\")\n",
" !{sys.executable} -m pip install -U sagemaker\n",
" IPython.Application.instance().kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.011722,
"end_time": "2021-06-01T00:13:00.200769",
"exception": false,
"start_time": "2021-06-01T00:13:00.189047",
"status": "completed"
},
"tags": []
},
"source": [
"## 1. Prepare training dataset\n",
"\n",
"### Tensorflow Datasets package\n",
"\n",
"First of all, set the notebook kernel to Tensorflow 2.x.\n",
"\n",
"We will use CIFAR-10 dataset for this experiment. To download CIFAR-10 datasets and convert it into TFRecord format, install `tensorflow-datasets` package, run `demo/generate_cifar10_tfrecords`, and upload tfrecord files to your S3 bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:00.228178Z",
"iopub.status.busy": "2021-06-01T00:13:00.227679Z",
"iopub.status.idle": "2021-06-01T00:13:22.616547Z",
"shell.execute_reply": "2021-06-01T00:13:22.616985Z"
},
"papermill": {
"duration": 22.404515,
"end_time": "2021-06-01T00:13:22.617136",
"exception": false,
"start_time": "2021-06-01T00:13:00.212621",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"!python demo/generate_cifar10_tfrecords.py --data-dir=./data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:22.761207Z",
"iopub.status.busy": "2021-06-01T00:13:22.760348Z",
"iopub.status.idle": "2021-06-01T00:13:27.334379Z",
"shell.execute_reply": "2021-06-01T00:13:27.334809Z"
},
"papermill": {
"duration": 4.663508,
"end_time": "2021-06-01T00:13:27.334960",
"exception": false,
"start_time": "2021-06-01T00:13:22.671452",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"import sagemaker\n",
"\n",
"s3_bucket = sagemaker.Session().default_bucket()\n",
"\n",
"dataset_prefix = \"data/cifar10-tfrecords\"\n",
"desired_s3_uri = f\"s3://{s3_bucket}/{dataset_prefix}\"\n",
"\n",
"dataset_location = sagemaker.s3.S3Uploader.upload(local_path=\"data\", desired_s3_uri=desired_s3_uri)\n",
"print(f\"Dataset uploaded to {dataset_location}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.051662,
"end_time": "2021-06-01T00:13:27.438637",
"exception": false,
"start_time": "2021-06-01T00:13:27.386975",
"status": "completed"
},
"tags": []
},
"source": [
"## 2. Create a Training Job with Profiling Enabled<a class=\"anchor\" id=\"option-1\"></a>\n",
"\n",
"We will use the standard [SageMaker Estimator API for Tensorflow](https://sagemaker.readthedocs.io/en/stable/frameworks/tensorflow/sagemaker.tensorflow.html#tensorflow-estimator) to create a training job. To enable profiling, we create a `ProfilerConfig` object and pass it to the `profiler_config` parameter of the `TensorFlow` estimator. For this demo, we set the the profiler to probe the system once every 500 miliseconds."
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.051632,
"end_time": "2021-06-01T00:13:27.541950",
"exception": false,
"start_time": "2021-06-01T00:13:27.490318",
"status": "completed"
},
"tags": []
},
"source": [
"### Set a profiler configuration"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:27.650304Z",
"iopub.status.busy": "2021-06-01T00:13:27.649576Z",
"iopub.status.idle": "2021-06-01T00:13:27.652036Z",
"shell.execute_reply": "2021-06-01T00:13:27.651534Z"
},
"papermill": {
"duration": 0.058237,
"end_time": "2021-06-01T00:13:27.652143",
"exception": false,
"start_time": "2021-06-01T00:13:27.593906",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"from sagemaker.debugger import ProfilerConfig, FrameworkProfile\n",
"\n",
"profiler_config = ProfilerConfig(\n",
" system_monitor_interval_millis=500,\n",
" framework_profile_params=FrameworkProfile(\n",
" local_path=\"/opt/ml/output/profiler/\", start_step=5, num_steps=2\n",
" ),\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.052106,
"end_time": "2021-06-01T00:13:27.756284",
"exception": false,
"start_time": "2021-06-01T00:13:27.704178",
"status": "completed"
},
"tags": []
},
"source": [
"### Configure Debugger hook\n",
"We configure the debugger hook to collect an excessive number of tensors, every 50 steps."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:27.865666Z",
"iopub.status.busy": "2021-06-01T00:13:27.864972Z",
"iopub.status.idle": "2021-06-01T00:13:27.866970Z",
"shell.execute_reply": "2021-06-01T00:13:27.867374Z"
},
"papermill": {
"duration": 0.058783,
"end_time": "2021-06-01T00:13:27.867516",
"exception": false,
"start_time": "2021-06-01T00:13:27.808733",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"import os\n",
"from sagemaker.debugger import DebuggerHookConfig, CollectionConfig\n",
"\n",
"debugger_hook_config = DebuggerHookConfig(\n",
" hook_parameters={\"save_interval\": \"50\"},\n",
" collection_configs=[\n",
" CollectionConfig(name=\"outputs\"),\n",
" CollectionConfig(name=\"gradients\"),\n",
" CollectionConfig(name=\"weights\"),\n",
" CollectionConfig(name=\"layers\"),\n",
" ],\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.052574,
"end_time": "2021-06-01T00:13:27.972348",
"exception": false,
"start_time": "2021-06-01T00:13:27.919774",
"status": "completed"
},
"tags": []
},
"source": [
"### Define hyperparameters\n",
"\n",
"The start-up script is set to [train_tf_bottleneck.py](./demo/train_tf_bottleneck.py). Define hyperparameters such as number of epochs, and batch size."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:28.083406Z",
"iopub.status.busy": "2021-06-01T00:13:28.082867Z",
"iopub.status.idle": "2021-06-01T00:13:28.084843Z",
"shell.execute_reply": "2021-06-01T00:13:28.085239Z"
},
"papermill": {
"duration": 0.058554,
"end_time": "2021-06-01T00:13:28.085393",
"exception": false,
"start_time": "2021-06-01T00:13:28.026839",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"hyperparameters = {\"epoch\": 2, \"batch_size\": 128}"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.052114,
"end_time": "2021-06-01T00:13:28.190111",
"exception": false,
"start_time": "2021-06-01T00:13:28.137997",
"status": "completed"
},
"tags": []
},
"source": [
"### Get the image URI\n",
"The image that we will is dependent on the region that you are running this notebook in."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:28.307641Z",
"iopub.status.busy": "2021-06-01T00:13:28.302079Z",
"iopub.status.idle": "2021-06-01T00:13:28.312840Z",
"shell.execute_reply": "2021-06-01T00:13:28.313237Z"
},
"papermill": {
"duration": 0.071134,
"end_time": "2021-06-01T00:13:28.313374",
"exception": false,
"start_time": "2021-06-01T00:13:28.242240",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"import boto3\n",
"\n",
"session = boto3.session.Session()\n",
"region = session.region_name\n",
"\n",
"image_uri = f\"763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-training:2.3.1-gpu-py37-cu110-ubuntu18.04\""
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.052806,
"end_time": "2021-06-01T00:13:28.419138",
"exception": false,
"start_time": "2021-06-01T00:13:28.366332",
"status": "completed"
},
"tags": []
},
"source": [
"### Define SageMaker Tensorflow Estimator\n",
"To enable profiling, you need to pass the Debugger profiling configuration (`profiler_config`), a list of Debugger rules (`rules`), and the image URI (`image_uri`) to the estimator. Debugger enables monitoring and profiling while the SageMaker estimator requests a training job."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:28.529298Z",
"iopub.status.busy": "2021-06-01T00:13:28.528797Z",
"iopub.status.idle": "2021-06-01T00:13:28.988436Z",
"shell.execute_reply": "2021-06-01T00:13:28.988897Z"
},
"papermill": {
"duration": 0.51737,
"end_time": "2021-06-01T00:13:28.989048",
"exception": false,
"start_time": "2021-06-01T00:13:28.471678",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"import sagemaker\n",
"from sagemaker.tensorflow import TensorFlow\n",
"\n",
"job_name = \"network-bottleneck\"\n",
"instance_count = 1\n",
"instance_type = \"ml.p2.xlarge\"\n",
"entry_script = \"train_tf_bottleneck.py\"\n",
"\n",
"estimator = TensorFlow(\n",
" role=sagemaker.get_execution_role(),\n",
" image_uri=image_uri,\n",
" base_job_name=job_name,\n",
" instance_type=instance_type,\n",
" instance_count=instance_count,\n",
" entry_point=entry_script,\n",
" source_dir=\"demo\",\n",
" profiler_config=profiler_config,\n",
" debugger_hook_config=debugger_hook_config,\n",
" script_mode=True,\n",
" hyperparameters=hyperparameters,\n",
" input_mode=\"Pipe\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.052871,
"end_time": "2021-06-01T00:13:29.095305",
"exception": false,
"start_time": "2021-06-01T00:13:29.042434",
"status": "completed"
},
"tags": []
},
"source": [
"> If you see an error, `TypeError: __init__() got an unexpected keyword argument 'instance_type'`, that means SageMaker Python SDK is out-dated. Please update your SageMaker Python SDK to 2.x by executing the below command and restart this notebook.\n",
"\n",
"```bash\n",
"pip install --upgrade sagemaker\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.052919,
"end_time": "2021-06-01T00:13:29.201150",
"exception": false,
"start_time": "2021-06-01T00:13:29.148231",
"status": "completed"
},
"tags": []
},
"source": [
"### Start training job\n",
"\n",
"The following `estimator.fit()` with `wait=False` argument initiates the training job in the background. You can proceed to run the dashboard or analysis notebooks."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:29.311413Z",
"iopub.status.busy": "2021-06-01T00:13:29.310918Z",
"iopub.status.idle": "2021-06-01T00:13:29.915090Z",
"shell.execute_reply": "2021-06-01T00:13:29.915520Z"
},
"papermill": {
"duration": 0.66101,
"end_time": "2021-06-01T00:13:29.915670",
"exception": false,
"start_time": "2021-06-01T00:13:29.254660",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"remote_inputs = {\"train\": dataset_location + \"/train\"}\n",
"\n",
"estimator.fit(remote_inputs, wait=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.053011,
"end_time": "2021-06-01T00:13:30.022207",
"exception": false,
"start_time": "2021-06-01T00:13:29.969196",
"status": "completed"
},
"tags": []
},
"source": [
"## 3. Monitor the system resource utilization using SageMaker Studio\n",
"\n",
"SageMaker Studio provides the visualization tool for Sagemaker Debugger where you can find the analysis report and the system and framework resource utilization history.\n",
"\n",
"To access this information in SageMaker Studio, click on the last icon on the left to open `SageMaker Components and registries` and choose `Experiments and trials`. You will see the list of training jobs. Right click on the job you want to investigate shows a pop-up menu, then click on `Open Debugger for insights` which opens a new tab for SageMaker Debugger.\n",
"\n",
"There are two tabs, `Overview` and `Nodes`. `Overview` gives profiling summaries for quick review, and `Nodes` gives a detailed utilization information on all nodes."
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.05283,
"end_time": "2021-06-01T00:13:30.127965",
"exception": false,
"start_time": "2021-06-01T00:13:30.075135",
"status": "completed"
},
"tags": []
},
"source": [
"## 4. SageMaker Debugger profiling analysis utilities\n",
"We can use the profiling analysis utilities to gain deeper insights into what the source of the issue is.\n",
"For this step, we will rely on the bokeh and smdebug packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:30.237604Z",
"iopub.status.busy": "2021-06-01T00:13:30.236910Z",
"iopub.status.idle": "2021-06-01T00:13:40.333732Z",
"shell.execute_reply": "2021-06-01T00:13:40.334232Z"
},
"papermill": {
"duration": 10.153683,
"end_time": "2021-06-01T00:13:40.334396",
"exception": false,
"start_time": "2021-06-01T00:13:30.180713",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"! pip install bokeh==2.1.1\n",
"! pip install smdebug==1.0.3"
]
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": 0.064631,
"end_time": "2021-06-01T00:13:40.464137",
"exception": false,
"start_time": "2021-06-01T00:13:40.399506",
"status": "completed"
},
"tags": []
},
"source": [
"Use smdebug to extract gpu and framework metrics"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2021-06-01T00:13:40.603797Z",
"iopub.status.busy": "2021-06-01T00:13:40.603249Z",
"iopub.status.idle": "2021-06-01T00:13:42.983379Z",
"shell.execute_reply": "2021-06-01T00:13:42.982357Z"
},
"papermill": {
"duration": 2.455191,
"end_time": "2021-06-01T00:13:42.983612",
"exception": true,
"start_time": "2021-06-01T00:13:40.528421",
"status": "failed"
},
"tags": []
},
"outputs": [],
"source": [
"import boto3\n",
"from smdebug.profiler.analysis.notebook_utils.training_job import TrainingJob\n",
"from smdebug.profiler.analysis.utils.profiler_data_to_pandas import PandasFrame\n",
"\n",
"\n",
"training_job_name = estimator.latest_training_job.name\n",
"region = boto3.Session().region_name\n",
"\n",
"tj = TrainingJob(training_job_name, region)\n",
"\n",
"pf = PandasFrame(tj.profiler_s3_output_path)\n",
"\n",
"# extract gpu metrics\n",
"system_metrics_df = pf.get_all_system_metrics()\n",
"gpus = system_metrics_df[system_metrics_df[\"dimension\"] == \"GPUUtilization\"]\n",
"timestamps = gpus[\"timestamp_us\"].to_numpy()\n",
"values = gpus[\"value\"].to_numpy()\n",
"\n",
"# exctract framework metrics\n",
"framework_metrics_df = pf.get_all_framework_metrics(\n",
" selected_framework_metrics=[\"Step:ModeKeys.TRAIN\", \"Step:ModeKeys.GLOBAL\"]\n",
")\n",
"train_steps = framework_metrics_df[\n",
" framework_metrics_df[\"framework_metric\"].isin([\"Step:ModeKeys.TRAIN\", \"Step:ModeKeys.GLOBAL\"])\n",
"]\n",
"start_step = train_steps[\"start_time_us\"].to_numpy()\n",
"end_step = train_steps[\"end_time_us\"].to_numpy()\n",
"step_num = train_steps[\"step\"].to_numpy()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"papermill": {
"duration": null,
"end_time": null,
"exception": null,
"start_time": null,
"status": "pending"
},
"tags": []
},
"source": [
"Use bokeh to plot the gpu metrics and the training progression on the same graph. This enables us to correlate between the two. We can see that the drops in gpu utilization coincide with every 50th step, which are marked in yellow. These are precisely the steps in which we have chosen to capture all of the graph tensors.\n",
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"papermill": {
"duration": null,
"end_time": null,
"exception": null,
"start_time": null,
"status": "pending"
},
"tags": []
},
"outputs": [],
"source": [
"import numpy as np\n",
"from bokeh.models import ColumnDataSource, CustomJS, Div, HoverTool, HBar\n",
"from bokeh.models.glyphs import Circle, Line\n",
"from bokeh.plotting import figure, show\n",
"\n",
"plot = figure(\n",
" plot_height=400,\n",
" plot_width=1400,\n",
" x_range=(timestamps[0], timestamps[-1]),\n",
" y_range=(-1, 110),\n",
" tools=\"crosshair,xbox_select,pan,reset,save,xwheel_zoom\",\n",
")\n",
"x_range = plot.x_range\n",
"\n",
"plot.xgrid.visible = False\n",
"plot.ygrid.visible = False\n",
"\n",
"colors = np.where(step_num % 50 == 0, \"yellow\", \"purple\")\n",
"\n",
"# pad framework metrics to match length of system metrics\n",
"pad = values.size - step_num.size\n",
"source = ColumnDataSource(\n",
" data=dict(\n",
" x=timestamps,\n",
" y=values,\n",
" left=np.pad(start_step, (0, pad)),\n",
" right=np.pad(end_step, (0, pad)),\n",
" color=np.pad(colors, (0, pad)),\n",
" )\n",
")\n",
"\n",
"\n",
"callback = CustomJS(\n",
" args=dict(s1=source, div=Div(width=250, height=100, height_policy=\"fixed\")),\n",
" code=\"\"\"\n",
" console.log('Running CustomJS callback now.');\n",
" var inds = s1.selected.indices;\n",
" console.log(inds);\n",
" var line = \"<span style=float:left;clear:left;font_size=13px><b> Selected index range: [\" + Math.min.apply(Math,inds) + \",\" + Math.max.apply(Math,inds) + \"]</b></span>\\\\n\";\n",
" console.log(line)\n",
" var text = div.text.concat(line);\n",
" var lines = text.split(\"\\\\n\")\n",
" if (lines.length > 35)\n",
" lines.shift();\n",
" div.text = lines.join(\"\\\\n\");\"\"\",\n",
")\n",
"\n",
"plot.js_on_event(\"selectiongeometry\", callback)\n",
"\n",
"line = Line(x=\"x\", y=\"y\", line_color=\"white\")\n",
"circle = Circle(x=\"x\", y=\"y\", fill_alpha=0, line_width=0)\n",
"hbar = HBar(\n",
" y=105, height=5, right=\"right\", left=\"left\", fill_color=\"color\", line_cap=\"round\", line_width=0\n",
")\n",
"\n",
"\n",
"p = plot.add_glyph(source, line)\n",
"p = plot.add_glyph(source, circle)\n",
"p = plot.add_glyph(source, hbar)\n",
"\n",
"# create tooltip for hover tool\n",
"hover = HoverTool(renderers=[p], tooltips=[(\"index\", \"$index\"), (\"(x,y)\", \"($x, $y)\")])\n",
"\n",
"plot.xaxis.axis_label = \"Time in ms\"\n",
"plot.yaxis.axis_label = \"GPU Utilization\"\n",
"plot.add_tools(hover)\n",
"show(plot, notebook_handle=True)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notebook CI Test Results\n",
"\n",
"This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Environment (conda_tensorflow2_p36)",
"language": "python",
"name": "conda_tensorflow2_p36"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.10"
},
"papermill": {
"default_parameters": {},
"duration": 45.057723,
"end_time": "2021-06-01T00:13:44.310180",
"environment_variables": {},
"exception": true,
"input_path": "callback_bottleneck.ipynb",
"output_path": "/opt/ml/processing/output/callback_bottleneck-2021-06-01-00-09-21.ipynb",
"parameters": {
"kms_key": "arn:aws:kms:us-west-2:521695447989:key/6e9984db-50cf-4c7e-926c-877ec47a8b25"
},
"start_time": "2021-06-01T00:12:59.252457",
"version": "2.3.3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}