bindings/jupyter/notebooks/manifold.ipynb (340 lines of code) (raw):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Manifold\n",
"\n",
"Manifold is a model-agnostic visual debugging tool for machine learning.\n",
"\n",
"Understanding ML model performance and behavior is a non-trivial process, given intrinsic opacity of ML algorithms. Performance summary statistics such as AUC, RMSE... are not instructive enough for identifying what went wrong with a model or how to improve it.\n",
"\n",
"As a visual analytics tool, Manifold allows ML practitioners to look beyond overall summary metrics to detect which subset of data a model is inaccurately predicting. Manifold also explains the potential cause of poor model performance by surfacing the feature distribution difference between better and worse-performing subsets of data.\n",
"\n",
"## Usage"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b3aa66eec5194995973fe2381159ae89",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Manifold(props='{\"data\": {\"x\": [{\"feature_0\": 947, \"feature_1\": \"D\"}, {\"feature_0\": 652, \"feature_1\": \"A\"}, {\"…"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from mlvis import Manifold\n",
"import sys, json, math\n",
"from random import uniform\n",
"\n",
"def generate_random_categorical_value(categories):\n",
" return categories[int(math.floor(uniform(0, 1) * len(categories)))]\n",
" \n",
"num_instances = 100\n",
"categories = ['A', 'B', 'C', 'D']\n",
"domain = [1, 1000]\n",
"classes = ['true', 'false']\n",
"\n",
"x = [{'feature_0': math.floor(uniform(*domain)), \n",
" 'feature_1': generate_random_categorical_value(categories)} \n",
" for i in range(0, num_instances)]\n",
"\n",
"yPred = [0] * 3\n",
"for i in range(0, len(yPred)):\n",
" yPred[i] = [0] * num_instances\n",
" for j in range(0, num_instances):\n",
" d = uniform(0, 1)\n",
" yPred[i][j] = {\n",
" classes[0]: d,\n",
" classes[1]: 1 - d\n",
" }\n",
"\n",
"yTrue = [generate_random_categorical_value(classes) for i in range(0, num_instances)]\n",
"\n",
"Manifold(props={'data': {\n",
" 'x': x,\n",
" 'yPred': yPred,\n",
" 'yTrue': yTrue\n",
"}})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data Format\n",
"\n",
"```python\n",
"data = {\n",
" x: [...], # feature data\n",
" yPred: [[...], ...] # prediction data\n",
" yTrue: [...], # ground truth data\n",
"}\n",
"```\n",
"\n",
"Each element in these lists represents one data point in your evaluation dataset, and the order of data instances in `x`, `yPred` and `yTrue` should all match.\n",
"Recommended instance count for each of these datasets is 10000 - 15000. If you have a larger dataset that you want to analyze, a random subset of your data generally suffices to reveal the important patterns in it.\n",
"\n",
"##### `x` (list | numpy.ndarray | pandas.DataFrame, required): \n",
"A list/ndarray/data_frame of instances with features. Example (2 data instances):"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"x = [\n",
" {'feature_0': 21, 'feature_1': 'B'},\n",
" {'feature_0': 36, 'feature_1': 'A'}\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example with ndarray:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([{'feature_0': 21, 'feature_1': 'B'},\n",
" {'feature_0': 36, 'feature_1': 'A'}], dtype=object)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"x = np.array([\n",
" {'feature_0': 21, 'feature_1': 'B'},\n",
" {'feature_0': 36, 'feature_1': 'A'}\n",
"])\n",
"x"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example with data_frame:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" feature_0 feature_1\n",
"0 21 B\n",
"1 36 A\n"
]
}
],
"source": [
"import pandas as pd\n",
"x = pd.DataFrame([\n",
" {'feature_0': 21, 'feature_1': 'B'},\n",
" {'feature_0': 36, 'feature_1': 'A'}\n",
"])\n",
"print(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### `yPred` (list, required):\n",
"A list of list or data frames, each child list is a prediction array from one model for each data instance. Example (3 models, 2 data instances, 2 classes `['false', 'true']`):"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"yPred = [\n",
" [{'false': 0.1, 'true': 0.9}, {'false': 0.8, 'true': 0.2}],\n",
" [{'false': 0.3, 'true': 0.7}, {'false': 0.9, 'true': 0.1}],\n",
" [{'false': 0.6, 'true': 0.4}, {'false': 0.4, 'true': 0.6}]\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example with a list of data frame:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model 0 is:\n",
" false true\n",
"0 0.1 0.9\n",
"1 0.8 0.2\n",
"Model 1 is:\n",
" false true\n",
"0 0.3 0.7\n",
"1 0.9 0.1\n",
"Model 2 is:\n",
" false true\n",
"0 0.6 0.4\n",
"1 0.4 0.6\n"
]
}
],
"source": [
"import pandas as pd\n",
"yPred = [\n",
" pd.DataFrame([{'false': 0.1, 'true': 0.9}, {'false': 0.8, 'true': 0.2}]),\n",
" pd.DataFrame([{'false': 0.3, 'true': 0.7}, {'false': 0.9, 'true': 0.1}]),\n",
" pd.DataFrame([{'false': 0.6, 'true': 0.4}, {'false': 0.4, 'true': 0.6}])\n",
"]\n",
"for i, y in enumerate(yPred):\n",
" print('Model ' + str(i) + ' is:')\n",
" print(y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### `yTrue` (list | numpy.ndarray | pandas.DataFrame, required):\n",
"A list, ground truth for each data instance. Values must be numbers for regression model, must be strings that match object keys in `yPred` for classification models. Example (2 data instances, 2 classes ['false', 'true']):"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"yTrue = [\n",
" 'true',\n",
" 'false'\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example with ndarray:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['true', 'false'], dtype='<U5')"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"yTrue = np.array([\n",
" 'true',\n",
" 'false'\n",
"])\n",
"yTrue"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example with data_frame:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" 0\n",
"0 true\n",
"1 false\n"
]
}
],
"source": [
"import pandas as pd\n",
"yTrue = pd.DataFrame([\n",
" 'true',\n",
" 'false'\n",
"])\n",
"print(yTrue)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}