ManagedkdbInsights/torq/create_all.ipynb (1,640 lines of code) (raw):
{
"cells": [
{
"cell_type": "markdown",
"id": "28bea13b-67bd-4a0e-8eab-3b8ffd37259e",
"metadata": {},
"source": [
"# TorQ: Create Everything\n",
"This notebook will use the AWS boto3 APIs to create the needed resources for a TorQ based application. The notebook will first clone the relevant gihub code (TorQ and TorQ AMazon FinSpace Starter Pack) then proceed to create the necessary AWS resources. \n",
"\n",
"Once you have create all clusters, you can see how to query for data through the gateway, see the [pykx_query_all](pykx_query_all.ipynb) notebook\n",
"\n",
"To cleanup (delete) all resources, run the [delete_all](delete_all.ipynb) notebook.\n",
"\n",
"## AWS Resources Created\n",
"- Database \n",
"- Changeset to add data to database \n",
"- Scaling Group that will contain all clusters \n",
"- Shared Volume \n",
"- Dataview of database on the shared volume \n",
"- Clusters\n",
"\n",
"This notebook is based on the TorQ Amazon FinSpace starter pack but uses Scaling Groups and Shared Volumes for cost savings.\n",
"\n",
"[TorQ Amazon FinSpace Starter Pack](https://dataintellecttech.github.io/TorQ-Amazon-FinSpace-Starter-Pack/)\n",
"\n",
"### Branches\n",
"\n",
"**TorQ-Amazon-FinSpace-Starter-Pack**: v1.0.2 \n",
"**TorQ**: v5.0.3\n",
"\n",
"**Note**: For other branches, be sure to update the git clone statements below."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "6c036169-ee58-48d0-a20e-f95364a6adf7",
"metadata": {},
"outputs": [],
"source": [
"!rm -rf torq_app*.zip "
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "97bf936e-20f2-419b-9e20-320f00f4d6c2",
"metadata": {},
"outputs": [],
"source": [
"!rm -rf TorQ TorQ-Amazon-FinSpace-Starter-Pack"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "94703544-445a-4142-9c65-28618797d1e6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cloning into 'TorQ-Amazon-FinSpace-Starter-Pack'...\n",
"remote: Enumerating objects: 228, done.\u001b[K\n",
"remote: Counting objects: 100% (228/228), done.\u001b[K\n",
"remote: Compressing objects: 100% (181/181), done.\u001b[K\n",
"remote: Total 228 (delta 41), reused 181 (delta 37), pack-reused 0 (from 0)\u001b[K\n",
"Receiving objects: 100% (228/228), 35.79 MiB | 33.38 MiB/s, done.\n",
"Resolving deltas: 100% (41/41), done.\n"
]
}
],
"source": [
"!git -c advice.detachedHead=false clone --depth 1 --branch v1.0.2 https://github.com/DataIntellectTech/TorQ-Amazon-FinSpace-Starter-Pack.git "
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "fb69788a-4523-41d4-9134-cf5e117fbbbe",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cloning into 'TorQ'...\n",
"remote: Enumerating objects: 797, done.\u001b[K\n",
"remote: Counting objects: 100% (797/797), done.\u001b[K\n",
"remote: Compressing objects: 100% (550/550), done.\u001b[K\n",
"remote: Total 797 (delta 195), reused 588 (delta 174), pack-reused 0 (from 0)\u001b[K\n",
"Receiving objects: 100% (797/797), 5.63 MiB | 25.96 MiB/s, done.\n",
"Resolving deltas: 100% (195/195), done.\n"
]
}
],
"source": [
"!git -c advice.detachedHead=false clone --depth 1 --branch v5.0.3 https://github.com/DataIntellectTech/TorQ.git "
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "1b2b0ba1-3169-47d2-a445-243686a91cba",
"metadata": {},
"outputs": [],
"source": [
"!ln -sf ../finspace_torq.q TorQ-Amazon-FinSpace-Starter-Pack\n",
"# this is the one modification over what is in the starter-pack on github"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "0d5f1d4a-ed45-44e3-bf75-9bdb75fcddbb",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import subprocess\n",
"import boto3\n",
"import json\n",
"import datetime\n",
"\n",
"import pykx as kx\n",
"\n",
"from managed_kx import *\n",
"\n",
"from env import *\n",
"from config import *\n",
"\n",
"# ----------------------------------------------------------------\n",
"\n",
"TORQ_CODEBASE=\"TorQ\"\n",
"TORQ_FINSPACE_CODEBASE=\"TorQ-Amazon-FinSpace-Starter-Pack\"\n",
"\n",
"# Source data directory\n",
"SOURCE_DATA_DIR=f\"{TORQ_FINSPACE_CODEBASE}/hdb\"\n",
"\n",
"# Code directory\n",
"CODEBASE=\"torq_app\"\n",
"\n",
"# S3 Destinations\n",
"S3_CODE_PATH=\"code\"\n",
"S3_DATA_PATH=\"data\"\n",
"\n",
"NODE_TYPE=\"kx.sg.xlarge\"\n",
"\n",
"DATABASE_CONFIG=[{ \n",
" 'databaseName': DB_NAME,\n",
" 'dataviewName': DBVIEW_NAME\n",
" }]\n",
"CODE_CONFIG={ 's3Bucket': S3_BUCKET, 's3Key': f'{S3_CODE_PATH}/{CODEBASE}.zip' }\n",
"\n",
"NAS1_CONFIG= {\n",
" 'type': 'SSD_250',\n",
" 'size': 1200\n",
"}\n",
"\n",
"VPC_CONFIG={ \n",
" 'vpcId': VPC_ID,\n",
" 'securityGroupIds': SECURITY_GROUPS,\n",
" 'subnetIds': SUBNET_IDS,\n",
" 'ipAddressType': 'IP_V4' \n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "3cfe7d89-9f5d-4ceb-ac8c-1f5054a6f15a",
"metadata": {},
"outputs": [],
"source": [
"# Using credentials and create service client\n",
"session = boto3.Session()\n",
"\n",
"# create finspace client\n",
"client = session.client(service_name='finspace')"
]
},
{
"cell_type": "markdown",
"id": "8c3d4047-9583-4b09-b75d-98fd2ddd6c36",
"metadata": {},
"source": [
"# Create the Database\n",
"Create a database from the supplied data in hdb.tar.gz. "
]
},
{
"cell_type": "markdown",
"id": "8bf690f2-c465-4df8-90f5-1e3b808bb368",
"metadata": {},
"source": [
"## Stage HDB Data on S3\n",
"Using AWS cli, copy hdb to staging bucket"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "aca7f0d7-32cd-443b-b642-e7209b8516ef",
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" PRE 2014.04.21/\n",
" PRE 2014.04.22/\n",
" PRE 2014.04.23/\n",
"2024-12-03 19:44:34 57 sym\n"
]
},
{
"data": {
"text/plain": [
"0"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"S3_DEST=f\"s3://{S3_BUCKET}/{S3_DATA_PATH}/{SOURCE_DATA_DIR}/\"\n",
"\n",
"if AWS_ACCESS_KEY_ID is not None:\n",
" cp = f\"\"\"\n",
"export AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID}\n",
"export AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY}\n",
"export AWS_SESSION_TOKEN={AWS_SESSION_TOKEN}\n",
"\n",
"aws s3 sync --quiet --exclude .DS_Store {SOURCE_DATA_DIR} {S3_DEST}\n",
"aws s3 ls {S3_DEST}\n",
"\"\"\"\n",
"else:\n",
" cp = f\"\"\"\n",
"aws s3 sync --quiet --exclude .DS_Store {SOURCE_DATA_DIR} {S3_DEST}\n",
"aws s3 ls {S3_DEST}\n",
"\"\"\"\n",
" \n",
"# execute the S3 copy\n",
"os.system(cp)"
]
},
{
"cell_type": "markdown",
"id": "17c759c4-ee6c-45c5-a9f6-6acacea3a3be",
"metadata": {},
"source": [
"## Create Managed Database\n",
"Using the AWS APIs, create a managed database in Managed kdb Insights."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "d55bd8d3-5629-46f9-bc1f-47bb0308dc0a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CREATING Database: finspace-database\n",
"CREATED Database: finspace-database\n",
"{\n",
" \"createdTimestamp\": \"2024-12-03 19:44:35.065000+00:00\",\n",
" \"databaseArn\": \"arn:aws:finspace:us-east-1:829845998889:kxEnvironment/jlcenjvtkgzrdek2qqv7ic/kxDatabase/finspace-database\",\n",
" \"databaseName\": \"finspace-database\",\n",
" \"description\": \"Basictick kdb database\",\n",
" \"environmentId\": \"jlcenjvtkgzrdek2qqv7ic\",\n",
" \"lastModifiedTimestamp\": \"2024-12-03 19:44:35.065000+00:00\"\n",
"}\n"
]
}
],
"source": [
"# assume it exists\n",
"create_db=False\n",
"\n",
"try:\n",
" resp = client.get_kx_database(environmentId=ENV_ID, databaseName=DB_NAME)\n",
" resp.pop('ResponseMetadata', None)\n",
"except:\n",
" # does not exist, will create\n",
" create_db=True\n",
"\n",
"if create_db:\n",
" print(f\"CREATING Database: {DB_NAME}\")\n",
" resp = client.create_kx_database(environmentId=ENV_ID, databaseName=DB_NAME, description=\"Basictick kdb database\")\n",
" resp.pop('ResponseMetadata', None)\n",
"\n",
" print(f\"CREATED Database: {DB_NAME}\")\n",
"\n",
"print(json.dumps(resp,sort_keys=True,indent=4,default=str))"
]
},
{
"cell_type": "markdown",
"id": "26d1194e-0c04-49a3-a7e7-a1d23fcff0d9",
"metadata": {},
"source": [
"## Add HDB Data to Database\n",
"Add the data in the local hdb directory to the managed database using the changeset mechanism. The Data will be copied to S3 then ingested with the create-kx-changeset API."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "7e03d62c-655b-4c64-aa9e-aaa04455a42c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c_set_list = list_kx_changesets(client, environmentId=ENV_ID, databaseName=DB_NAME)\n",
"len(c_set_list)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "eae61f04-1c9c-468e-bb38-b2e0b94897a0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Changeset...\n",
"{\n",
" \"changeRequests\": [\n",
" {\n",
" \"changeType\": \"PUT\",\n",
" \"dbPath\": \"/2014.04.23/\",\n",
" \"s3Path\": \"s3://kdb-demo-829845998889-kms/data/TorQ-Amazon-FinSpace-Starter-Pack/hdb/2014.04.23/\"\n",
" },\n",
" {\n",
" \"changeType\": \"PUT\",\n",
" \"dbPath\": \"/2014.04.22/\",\n",
" \"s3Path\": \"s3://kdb-demo-829845998889-kms/data/TorQ-Amazon-FinSpace-Starter-Pack/hdb/2014.04.22/\"\n",
" },\n",
" {\n",
" \"changeType\": \"PUT\",\n",
" \"dbPath\": \"/\",\n",
" \"s3Path\": \"s3://kdb-demo-829845998889-kms/data/TorQ-Amazon-FinSpace-Starter-Pack/hdb/sym\"\n",
" },\n",
" {\n",
" \"changeType\": \"PUT\",\n",
" \"dbPath\": \"/2014.04.21/\",\n",
" \"s3Path\": \"s3://kdb-demo-829845998889-kms/data/TorQ-Amazon-FinSpace-Starter-Pack/hdb/2014.04.21/\"\n",
" }\n",
" ],\n",
" \"changesetId\": \"2MnHBswHfdRpfkJBJVMANw\",\n",
" \"createdTimestamp\": \"2024-12-03 19:44:35.856000+00:00\",\n",
" \"databaseName\": \"finspace-database\",\n",
" \"environmentId\": \"jlcenjvtkgzrdek2qqv7ic\",\n",
" \"lastModifiedTimestamp\": \"2024-12-03 19:44:35.856000+00:00\",\n",
" \"status\": \"PENDING\"\n",
"}\n"
]
}
],
"source": [
"# TODO: check is there is a changeset in the database, if so, no need to add another\n",
"c_set_list = list_kx_changesets(client, environmentId=ENV_ID, databaseName=DB_NAME)\n",
"\n",
"if len(c_set_list) == 0:\n",
" # if changesets exist, set chagnset_id to last created one\n",
"\n",
" changes=[]\n",
"\n",
" for f in os.listdir(f\"{SOURCE_DATA_DIR}\"):\n",
" if os.path.isdir(f\"{SOURCE_DATA_DIR}/{f}\"):\n",
" changes.append( { 'changeType': 'PUT', 's3Path': f\"{S3_DEST}{f}/\", 'dbPath': f\"/{f}/\" } )\n",
" else:\n",
" changes.append( { 'changeType': 'PUT', 's3Path': f\"{S3_DEST}{f}\", 'dbPath': f\"/\" } )\n",
"\n",
" resp = client.create_kx_changeset(environmentId=ENV_ID, databaseName=DB_NAME, \n",
" changeRequests=changes)\n",
"\n",
" resp.pop('ResponseMetadata', None)\n",
" changeset_id = resp['changesetId']\n",
"\n",
" print(\"Changeset...\")\n",
" print(json.dumps(resp,sort_keys=True,indent=4,default=str))\n",
"else:\n",
" c_set_list=sorted(c_set_list, key=lambda d: d['createdTimestamp']) \n",
" changeset_id=c_set_list[-1]['changesetId']\n",
" print(f\"Using Last changeset: {changeset_id}\")\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "b4422bdd-7d44-4fb0-8018-0bebd6987704",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Status is IN_PROGRESS, total wait 0:00:00, waiting 10 sec ...\n",
"Status is IN_PROGRESS, total wait 0:00:10, waiting 10 sec ...\n",
"Status is IN_PROGRESS, total wait 0:00:20, waiting 10 sec ...\n",
"**Done**\n"
]
}
],
"source": [
"wait_for_changeset_status(client, environmentId=ENV_ID, databaseName=DB_NAME, changesetId=changeset_id, show_wait=True)\n",
"print(\"**Done**\")"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "8ba008f3-4991-474c-9b3e-43a1dca56257",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"====================================================================================================\n",
"Database: finspace-database, Changesets: 1 \n",
"====================================================================================================\n",
" Changeset: 2MnHBswHfdRpfkJBJVMANw: Created: 2024-12-03 19:44:35.856000+00:00 (COMPLETED)\n"
]
},
{
"data": {
"text/html": [
"<style type=\"text/css\">\n",
"</style>\n",
"<table id=\"T_4f9eb\">\n",
" <thead>\n",
" <tr>\n",
" <th id=\"T_4f9eb_level0_col0\" class=\"col_heading level0 col0\" >changeType</th>\n",
" <th id=\"T_4f9eb_level0_col1\" class=\"col_heading level0 col1\" >s3Path</th>\n",
" <th id=\"T_4f9eb_level0_col2\" class=\"col_heading level0 col2\" >dbPath</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td id=\"T_4f9eb_row0_col0\" class=\"data row0 col0\" >PUT</td>\n",
" <td id=\"T_4f9eb_row0_col1\" class=\"data row0 col1\" >s3://kdb-demo-829845998889-kms/data/TorQ-Amazon-FinSpace-Starter-Pack/hdb/2014.04.23/</td>\n",
" <td id=\"T_4f9eb_row0_col2\" class=\"data row0 col2\" >/2014.04.23/</td>\n",
" </tr>\n",
" <tr>\n",
" <td id=\"T_4f9eb_row1_col0\" class=\"data row1 col0\" >PUT</td>\n",
" <td id=\"T_4f9eb_row1_col1\" class=\"data row1 col1\" >s3://kdb-demo-829845998889-kms/data/TorQ-Amazon-FinSpace-Starter-Pack/hdb/2014.04.22/</td>\n",
" <td id=\"T_4f9eb_row1_col2\" class=\"data row1 col2\" >/2014.04.22/</td>\n",
" </tr>\n",
" <tr>\n",
" <td id=\"T_4f9eb_row2_col0\" class=\"data row2 col0\" >PUT</td>\n",
" <td id=\"T_4f9eb_row2_col1\" class=\"data row2 col1\" >s3://kdb-demo-829845998889-kms/data/TorQ-Amazon-FinSpace-Starter-Pack/hdb/sym</td>\n",
" <td id=\"T_4f9eb_row2_col2\" class=\"data row2 col2\" >/</td>\n",
" </tr>\n",
" <tr>\n",
" <td id=\"T_4f9eb_row3_col0\" class=\"data row3 col0\" >PUT</td>\n",
" <td id=\"T_4f9eb_row3_col1\" class=\"data row3 col1\" >s3://kdb-demo-829845998889-kms/data/TorQ-Amazon-FinSpace-Starter-Pack/hdb/2014.04.21/</td>\n",
" <td id=\"T_4f9eb_row3_col2\" class=\"data row3 col2\" >/2014.04.21/</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n"
],
"text/plain": [
"<pandas.io.formats.style.Styler at 0x7f8b6519f010>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"note_str = \"\"\n",
"\n",
"c_set_list = list_kx_changesets(client, environmentId=ENV_ID, databaseName=DB_NAME)\n",
"\n",
"if len(c_set_list) == 0:\n",
" note_str = \"<<Could not get changesets>>\"\n",
" \n",
"print(100*\"=\")\n",
"print(f\"Database: {DB_NAME}, Changesets: {len(c_set_list)} {note_str}\")\n",
"print(100*\"=\")\n",
"\n",
"# sort by create time\n",
"c_set_list = sorted(c_set_list, key=lambda d: d['createdTimestamp']) \n",
"\n",
"for c in c_set_list:\n",
" c_set_id = c['changesetId']\n",
" print(f\" Changeset: {c_set_id}: Created: {c['createdTimestamp']} ({c['status']})\")\n",
" c_rqs = client.get_kx_changeset(environmentId=ENV_ID, databaseName=DB_NAME, changesetId=c_set_id)['changeRequests']\n",
"\n",
" chs_pdf = pd.DataFrame.from_dict(c_rqs).style.hide(axis='index')\n",
" display(chs_pdf)"
]
},
{
"cell_type": "markdown",
"id": "9dae0232-3666-491f-8891-dae30e12c9d8",
"metadata": {},
"source": [
"# Create Scaling Group\n",
"The scaling group represents the total compute avilable to the application. All clusters will be placed into the scaling group ans share the compute and memory of the scaling group."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "613be7f8-fb82-4415-b30c-186ed470dba4",
"metadata": {},
"outputs": [],
"source": [
"# Check if scaling group exits, only create if it does not\n",
"resp = get_kx_scaling_group(client=client, environmentId=ENV_ID, scalingGroupName=SCALING_GROUP_NAME)\n",
"\n",
"if resp is None:\n",
" resp = client.create_kx_scaling_group(\n",
" environmentId = ENV_ID, \n",
" scalingGroupName = SCALING_GROUP_NAME,\n",
" hostType=NODE_TYPE,\n",
" availabilityZoneId = AZ_ID\n",
" )\n",
"else:\n",
" print(f\"Scaling Group {SCALING_GROUP_NAME} exists\")"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "cab8d287-4c24-4e53-8dbd-642c73c7faf5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'ResponseMetadata': {'RequestId': '8fae45fa-a679-4d72-bf5c-d08a4f8a902b',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'application/json',\n",
" 'content-length': '237',\n",
" 'connection': 'keep-alive',\n",
" 'date': 'Tue, 03 Dec 2024 19:45:11 GMT',\n",
" 'x-amzn-requestid': '8fae45fa-a679-4d72-bf5c-d08a4f8a902b',\n",
" 'x-amz-apigw-id': 'COvm1ErOIAMEG2Q=',\n",
" 'x-amzn-trace-id': 'Root=1-674f5fc4-3ca0c0cc723507f97a1f2064',\n",
" 'x-cache': 'Miss from cloudfront',\n",
" 'via': '1.1 2174e600dd54879ba9f49d0337eeb2dc.cloudfront.net (CloudFront)',\n",
" 'x-amz-cf-pop': 'IAD55-P1',\n",
" 'x-amz-cf-id': 'nMZmwBT6_ZZvmJ_VtVOzfZJCQaYULYTwjCr_uyptzofKaxnns6VJig=='},\n",
" 'RetryAttempts': 0},\n",
" 'environmentId': 'jlcenjvtkgzrdek2qqv7ic',\n",
" 'scalingGroupName': 'SCALING_GROUP_torq',\n",
" 'hostType': 'kx.sg.xlarge',\n",
" 'availabilityZoneId': 'use1-az6',\n",
" 'status': 'CREATING',\n",
" 'lastModifiedTimestamp': datetime.datetime(2024, 12, 3, 19, 45, 10, 883000, tzinfo=tzlocal()),\n",
" 'createdTimestamp': datetime.datetime(2024, 12, 3, 19, 45, 10, 813000, tzinfo=tzlocal())}"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"resp"
]
},
{
"cell_type": "markdown",
"id": "6943fb16-8989-4199-a0fd-5c7c0d5aa56e",
"metadata": {},
"source": [
"# Create Shared Volume\n",
"The shared volume is a common storage device for the application. Every cluster using the shared volume will have a writable directory named after the cluster, can read the directories named after other clusters in the application using the volume. Also, there is a common "
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "e4a8a247-d029-4f9b-aaf5-c6e2ffe200a1",
"metadata": {},
"outputs": [],
"source": [
"# Check if volume already exists before trying to create one\n",
"resp = get_kx_volume(client=client, environmentId=ENV_ID, volumeName=VOLUME_NAME)\n",
"\n",
"if resp is None:\n",
" resp = client.create_kx_volume(\n",
" environmentId = ENV_ID, \n",
" volumeType = 'NAS_1',\n",
" volumeName = VOLUME_NAME,\n",
" description = 'Shared volume between TP and RDB',\n",
" nas1Configuration = NAS1_CONFIG,\n",
" azMode='SINGLE',\n",
" availabilityZoneIds=[ AZ_ID ] \n",
" )\n",
"else:\n",
" print(f\"Volume {VOLUME_NAME} exists\") "
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "3ec4876f-5ecc-420e-aff8-9c88c8c9deb4",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'ResponseMetadata': {'RequestId': 'eaf7e431-4f46-4533-95e1-f421588144b7',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'application/json',\n",
" 'content-length': '431',\n",
" 'connection': 'keep-alive',\n",
" 'date': 'Tue, 03 Dec 2024 19:45:15 GMT',\n",
" 'x-amzn-requestid': 'eaf7e431-4f46-4533-95e1-f421588144b7',\n",
" 'x-amz-apigw-id': 'COvnTHSKoAMEaGg=',\n",
" 'x-amzn-trace-id': 'Root=1-674f5fc7-572cf2f6707f24e716735df3',\n",
" 'x-cache': 'Miss from cloudfront',\n",
" 'via': '1.1 2174e600dd54879ba9f49d0337eeb2dc.cloudfront.net (CloudFront)',\n",
" 'x-amz-cf-pop': 'IAD55-P1',\n",
" 'x-amz-cf-id': 'le2T_WMpJHOtSEkdHcKMALw1tlpVU4kHxJar2FyoxVvh6v_MQkKPLw=='},\n",
" 'RetryAttempts': 0},\n",
" 'environmentId': 'jlcenjvtkgzrdek2qqv7ic',\n",
" 'volumeName': 'SHARED_torq',\n",
" 'volumeType': 'NAS_1',\n",
" 'volumeArn': 'arn:aws:finspace:us-east-1:829845998889:kxEnvironment/jlcenjvtkgzrdek2qqv7ic/kxVolume/SHARED_torq',\n",
" 'nas1Configuration': {'type': 'SSD_250', 'size': 1200},\n",
" 'status': 'CREATING',\n",
" 'azMode': 'SINGLE',\n",
" 'description': 'Shared volume between TP and RDB',\n",
" 'availabilityZoneIds': ['use1-az6'],\n",
" 'createdTimestamp': datetime.datetime(2024, 12, 3, 19, 45, 14, 29000, tzinfo=tzlocal())}"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"resp"
]
},
{
"cell_type": "markdown",
"id": "9e718537-8853-4f21-8cc7-36b489e380c4",
"metadata": {},
"source": [
"# Wait for Volume and Scaling Group\n",
"Before proceeding to use Volumes and Scaling groups, wait for their creation to complete.\n",
"\n",
"Volume will be used by the dataview. \n",
"Dataview and Scaling Group will be used by the clusters\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "a3ed8931-e458-4ffb-83cc-0aa4da4d9f98",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:00:00, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:00:30, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:01:00, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:01:30, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:02:00, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:02:30, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:03:00, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:03:30, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:04:00, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is CREATING, total wait 0:04:30, waiting 30 sec ...\n",
"Scaling Group: SCALING_GROUP_torq status is now ACTIVE, total wait 0:05:00\n",
"** DONE **\n",
"Volume: SHARED_torq status is CREATING, total wait 0:00:00, waiting 30 sec ...\n",
"Volume: SHARED_torq status is CREATING, total wait 0:00:30, waiting 30 sec ...\n",
"Volume: SHARED_torq status is CREATING, total wait 0:01:00, waiting 30 sec ...\n",
"Volume: SHARED_torq status is CREATING, total wait 0:01:30, waiting 30 sec ...\n",
"Volume: SHARED_torq status is CREATING, total wait 0:02:00, waiting 30 sec ...\n",
"Volume: SHARED_torq status is CREATING, total wait 0:02:30, waiting 30 sec ...\n",
"Volume: SHARED_torq status is CREATING, total wait 0:03:00, waiting 30 sec ...\n",
"Volume: SHARED_torq status is CREATING, total wait 0:03:30, waiting 30 sec ...\n",
"Volume: SHARED_torq status is CREATING, total wait 0:04:00, waiting 30 sec ...\n",
"Volume: SHARED_torq status is now ACTIVE, total wait 0:04:30\n",
"** DONE **\n"
]
}
],
"source": [
"# wait for the scaling group to create\n",
"wait_for_scaling_group_status(client=client, environmentId=ENV_ID, scalingGroupName=SCALING_GROUP_NAME, show_wait=True)\n",
"print(\"** DONE **\")\n",
"\n",
"# wait for the volume to create\n",
"wait_for_volume_status(client=client, environmentId=ENV_ID, volumeName=VOLUME_NAME, show_wait=True)\n",
"print(\"** DONE **\")"
]
},
{
"cell_type": "markdown",
"id": "fe41eaeb-9c8e-44d3-b8bc-f354142f9140",
"metadata": {},
"source": [
"# Create Dataview\n",
"Create a dataview, for a specific (static) version of the database and have all of its data cached using the shared volume."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "03434316-4ccc-420d-adee-715e6eb1bcd6",
"metadata": {},
"outputs": [],
"source": [
"# Check if dataview already exists and is set to the requested changeset_id\n",
"resp = get_kx_dataview(client=client, environmentId=ENV_ID, databaseName=DB_NAME, dataviewName=DBVIEW_NAME)\n",
"\n",
"if resp is None:\n",
" # sort changeset list by create time\n",
" c_set_list = sorted(c_set_list, key=lambda d: d['createdTimestamp']) \n",
"\n",
" resp = client.create_kx_dataview(\n",
" environmentId = ENV_ID, \n",
" databaseName=DB_NAME, \n",
" dataviewName=DBVIEW_NAME,\n",
" azMode='SINGLE',\n",
" availabilityZoneId=AZ_ID,\n",
" changesetId=c_set_list[-1]['changesetId'],\n",
" segmentConfigurations=[\n",
" { \n",
" 'dbPaths': ['/*'],\n",
" 'volumeName': VOLUME_NAME\n",
" }\n",
" ],\n",
" autoUpdate=False,\n",
" description = f'Dataview of database'\n",
" )\n",
"else:\n",
" print(f\"Dataview {DBVIEW_NAME} exists\") "
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "734ab7da-ef78-4701-9a58-1eeacbd9c557",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:00:00, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:00:30, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:01:00, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:01:30, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:02:00, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:02:30, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:03:00, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:03:30, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:04:00, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:04:30, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:05:00, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:05:30, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is CREATING, total wait 0:06:00, waiting 30 sec ...\n",
"Dataview: finspace-database_DBVIEW status is now ACTIVE, total wait 0:06:30\n",
"** DONE **\n"
]
}
],
"source": [
"# wait for the view to create\n",
"wait_for_dataview_status(client=client, environmentId=ENV_ID, databaseName=DB_NAME, dataviewName=DBVIEW_NAME, show_wait=True)\n",
"print(\"** DONE **\")"
]
},
{
"cell_type": "markdown",
"id": "dea431b0-c501-46bb-b72a-a5eb80a335b0",
"metadata": {},
"source": [
"# Create Clusters\n",
"With foundation resources now completed, create the needed clusters for the application."
]
},
{
"cell_type": "markdown",
"id": "0f0c06ab-4dcb-4cc6-abc9-2c77ff3a4242",
"metadata": {
"tags": []
},
"source": [
"## Stage Code to S3\n",
"Code to be used in this application must be staged to an S3 bucket the service can read from, that code will then be deployed to the clusters as part of their creation workflow."
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "b502a0a5-8610-4fc8-b6b7-04c47e89ba75",
"metadata": {
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"upload: ./torq_app.zip to s3://kdb-demo-829845998889-kms/code/torq_app.zip\n",
"2024-12-03 20:01:30 486888 torq_app.zip\n"
]
},
{
"data": {
"text/plain": [
"0"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# zip the code\n",
"os.system(f\"zip -q -r {CODEBASE}.zip {TORQ_CODEBASE}/ {TORQ_FINSPACE_CODEBASE}/ -x '*.ipynb_checkpoints*' -x '*/hdb/*' -x '*.git*' -x '*/tests/*' -x '*/terraform-deployment/*' -x '*/docs/*' -x '*/lib/*' -x '*/html/*' -x '*/datadog/*' -x '*/monit/*'\")\n",
"\n",
"cp = \"\"\n",
"\n",
"# copy code to S3\n",
"if AWS_ACCESS_KEY_ID is not None:\n",
" cp = f\"\"\"\n",
"export AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID}\n",
"export AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY}\n",
"export AWS_SESSION_TOKEN={AWS_SESSION_TOKEN}\n",
"\"\"\"\n",
"\n",
"cp += f\"\"\"\n",
"aws s3 cp --exclude .DS_Store {CODEBASE}.zip s3://{S3_BUCKET}/code/{CODEBASE}.zip\n",
"aws s3 ls s3://{S3_BUCKET}/code/{CODEBASE}.zip\n",
"\"\"\"\n",
"\n",
"# execute the S3 copy\n",
"os.system(cp)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "d0a75d53-e567-4a2d-b47b-170829d79f63",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating: discovery1\n"
]
},
{
"data": {
"text/plain": [
"{'ResponseMetadata': {'RequestId': '9707ab7a-a90c-4301-a924-7a89f11665ea',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'application/json',\n",
" 'content-length': '1675',\n",
" 'connection': 'keep-alive',\n",
" 'date': 'Tue, 03 Dec 2024 20:01:36 GMT',\n",
" 'x-amzn-requestid': '9707ab7a-a90c-4301-a924-7a89f11665ea',\n",
" 'x-amz-apigw-id': 'COyAKG5KoAMEqUA=',\n",
" 'x-amzn-trace-id': 'Root=1-674f639a-3d8d59df232d653156b73bcb',\n",
" 'x-cache': 'Miss from cloudfront',\n",
" 'via': '1.1 2174e600dd54879ba9f49d0337eeb2dc.cloudfront.net (CloudFront)',\n",
" 'x-amz-cf-pop': 'IAD55-P1',\n",
" 'x-amz-cf-id': 'FJtqQW8JhAuRCTkhuP_HW1RfVg0YKWQk3Qh4Gu8pAqnV1NXNSebsGQ=='},\n",
" 'RetryAttempts': 0},\n",
" 'status': 'PENDING',\n",
" 'clusterName': 'discovery1',\n",
" 'clusterType': 'GP',\n",
" 'volumes': [{'volumeName': 'SHARED_torq', 'volumeType': 'NAS_1'}],\n",
" 'databases': [{'databaseName': 'finspace-database',\n",
" 'cacheConfigurations': [],\n",
" 'dataviewConfiguration': {'dataviewName': 'finspace-database_DBVIEW',\n",
" 'dataviewVersionId': 'jsnHC4G7CSYi9LS9fKwUNw',\n",
" 'changesetId': '2MnHBswHfdRpfkJBJVMANw',\n",
" 'segmentConfigurations': [{'dbPaths': ['/*'],\n",
" 'volumeName': 'SHARED_torq',\n",
" 'onDemand': False}]}}],\n",
" 'clusterDescription': 'Created with create_all notebook',\n",
" 'releaseLabel': '1.0',\n",
" 'vpcConfiguration': {'vpcId': 'vpc-0fe2b9c50f3ad382f',\n",
" 'securityGroupIds': ['sg-0c99f1cfb9c3c7fd9'],\n",
" 'subnetIds': ['subnet-04052219ec25b062b'],\n",
" 'ipAddressType': 'IP_V4'},\n",
" 'initializationScript': 'TorQ-Amazon-FinSpace-Starter-Pack/finspace_torq.q',\n",
" 'commandLineArguments': [{'key': 'proctype', 'value': 'discovery'},\n",
" {'key': 'procname', 'value': 'discovery1'},\n",
" {'key': 'jsonlogs', 'value': 'true'},\n",
" {'key': 'noredirect', 'value': 'true'},\n",
" {'key': 's', 'value': '2'}],\n",
" 'code': {'s3Bucket': 'kdb-demo-829845998889-kms',\n",
" 's3Key': 'code/torq_app.zip'},\n",
" 'executionRole': 'arn:aws:iam::829845998889:role/kdb-all-user',\n",
" 'lastModifiedTimestamp': datetime.datetime(2024, 12, 3, 20, 1, 35, 774000, tzinfo=tzlocal()),\n",
" 'savedownStorageConfiguration': {'volumeName': 'SHARED_torq'},\n",
" 'azMode': 'SINGLE',\n",
" 'availabilityZoneId': 'use1-az6',\n",
" 'createdTimestamp': datetime.datetime(2024, 12, 3, 20, 1, 35, 746000, tzinfo=tzlocal()),\n",
" 'scalingGroupConfiguration': {'scalingGroupName': 'SCALING_GROUP_torq',\n",
" 'memoryReservation': 6,\n",
" 'nodeCount': 1}}"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating: rdb1\n"
]
},
{
"data": {
"text/plain": [
"{'ResponseMetadata': {'RequestId': '5d143ee5-9ac9-4f66-b20f-5d962a20ca39',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'application/json',\n",
" 'content-length': '1657',\n",
" 'connection': 'keep-alive',\n",
" 'date': 'Tue, 03 Dec 2024 20:01:39 GMT',\n",
" 'x-amzn-requestid': '5d143ee5-9ac9-4f66-b20f-5d962a20ca39',\n",
" 'x-amz-apigw-id': 'COyBIF2ooAMEtrA=',\n",
" 'x-amzn-trace-id': 'Root=1-674f63a0-5b7cf8f1694fdafe0ddd9e87',\n",
" 'x-cache': 'Miss from cloudfront',\n",
" 'via': '1.1 2174e600dd54879ba9f49d0337eeb2dc.cloudfront.net (CloudFront)',\n",
" 'x-amz-cf-pop': 'IAD55-P1',\n",
" 'x-amz-cf-id': '77h8oE1aIZrLTFjB-DhJPd7nmC1wVIbXBMlKkqWQhSzKTXfEg1cbgg=='},\n",
" 'RetryAttempts': 0},\n",
" 'status': 'PENDING',\n",
" 'clusterName': 'rdb1',\n",
" 'clusterType': 'GP',\n",
" 'volumes': [{'volumeName': 'SHARED_torq', 'volumeType': 'NAS_1'}],\n",
" 'databases': [{'databaseName': 'finspace-database',\n",
" 'cacheConfigurations': [],\n",
" 'dataviewConfiguration': {'dataviewName': 'finspace-database_DBVIEW',\n",
" 'dataviewVersionId': 'jsnHC4G7CSYi9LS9fKwUNw',\n",
" 'changesetId': '2MnHBswHfdRpfkJBJVMANw',\n",
" 'segmentConfigurations': [{'dbPaths': ['/*'],\n",
" 'volumeName': 'SHARED_torq',\n",
" 'onDemand': False}]}}],\n",
" 'clusterDescription': 'Created with create_all notebook',\n",
" 'releaseLabel': '1.0',\n",
" 'vpcConfiguration': {'vpcId': 'vpc-0fe2b9c50f3ad382f',\n",
" 'securityGroupIds': ['sg-0c99f1cfb9c3c7fd9'],\n",
" 'subnetIds': ['subnet-04052219ec25b062b'],\n",
" 'ipAddressType': 'IP_V4'},\n",
" 'initializationScript': 'TorQ-Amazon-FinSpace-Starter-Pack/finspace_torq.q',\n",
" 'commandLineArguments': [{'key': 'proctype', 'value': 'rdb'},\n",
" {'key': 'procname', 'value': 'rdb1'},\n",
" {'key': 'jsonlogs', 'value': 'true'},\n",
" {'key': 'noredirect', 'value': 'true'},\n",
" {'key': 's', 'value': '2'}],\n",
" 'code': {'s3Bucket': 'kdb-demo-829845998889-kms',\n",
" 's3Key': 'code/torq_app.zip'},\n",
" 'executionRole': 'arn:aws:iam::829845998889:role/kdb-all-user',\n",
" 'lastModifiedTimestamp': datetime.datetime(2024, 12, 3, 20, 1, 39, 16000, tzinfo=tzlocal()),\n",
" 'savedownStorageConfiguration': {'volumeName': 'SHARED_torq'},\n",
" 'azMode': 'SINGLE',\n",
" 'availabilityZoneId': 'use1-az6',\n",
" 'createdTimestamp': datetime.datetime(2024, 12, 3, 20, 1, 39, 7000, tzinfo=tzlocal()),\n",
" 'scalingGroupConfiguration': {'scalingGroupName': 'SCALING_GROUP_torq',\n",
" 'memoryReservation': 6,\n",
" 'nodeCount': 1}}"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating: hdb1\n"
]
},
{
"data": {
"text/plain": [
"{'ResponseMetadata': {'RequestId': 'bc019e1c-69b3-44c7-be2c-f6d7549b65ec',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'application/json',\n",
" 'content-length': '1657',\n",
" 'connection': 'keep-alive',\n",
" 'date': 'Tue, 03 Dec 2024 20:01:41 GMT',\n",
" 'x-amzn-requestid': 'bc019e1c-69b3-44c7-be2c-f6d7549b65ec',\n",
" 'x-amz-apigw-id': 'COyBlFYsoAMEFZQ=',\n",
" 'x-amzn-trace-id': 'Root=1-674f63a3-1f1f1b980804c82e4d1c6bf5',\n",
" 'x-cache': 'Miss from cloudfront',\n",
" 'via': '1.1 2174e600dd54879ba9f49d0337eeb2dc.cloudfront.net (CloudFront)',\n",
" 'x-amz-cf-pop': 'IAD55-P1',\n",
" 'x-amz-cf-id': 'nXuom7ncN8vqDJeMGJNjZHzfZetheQX7hc3JhaxcqH6RgLQp5Sb2bw=='},\n",
" 'RetryAttempts': 0},\n",
" 'status': 'PENDING',\n",
" 'clusterName': 'hdb1',\n",
" 'clusterType': 'GP',\n",
" 'volumes': [{'volumeName': 'SHARED_torq', 'volumeType': 'NAS_1'}],\n",
" 'databases': [{'databaseName': 'finspace-database',\n",
" 'cacheConfigurations': [],\n",
" 'dataviewConfiguration': {'dataviewName': 'finspace-database_DBVIEW',\n",
" 'dataviewVersionId': 'jsnHC4G7CSYi9LS9fKwUNw',\n",
" 'changesetId': '2MnHBswHfdRpfkJBJVMANw',\n",
" 'segmentConfigurations': [{'dbPaths': ['/*'],\n",
" 'volumeName': 'SHARED_torq',\n",
" 'onDemand': False}]}}],\n",
" 'clusterDescription': 'Created with create_all notebook',\n",
" 'releaseLabel': '1.0',\n",
" 'vpcConfiguration': {'vpcId': 'vpc-0fe2b9c50f3ad382f',\n",
" 'securityGroupIds': ['sg-0c99f1cfb9c3c7fd9'],\n",
" 'subnetIds': ['subnet-04052219ec25b062b'],\n",
" 'ipAddressType': 'IP_V4'},\n",
" 'initializationScript': 'TorQ-Amazon-FinSpace-Starter-Pack/finspace_torq.q',\n",
" 'commandLineArguments': [{'key': 'proctype', 'value': 'hdb'},\n",
" {'key': 'procname', 'value': 'hdb1'},\n",
" {'key': 'jsonlogs', 'value': 'true'},\n",
" {'key': 'noredirect', 'value': 'true'},\n",
" {'key': 's', 'value': '4'}],\n",
" 'code': {'s3Bucket': 'kdb-demo-829845998889-kms',\n",
" 's3Key': 'code/torq_app.zip'},\n",
" 'executionRole': 'arn:aws:iam::829845998889:role/kdb-all-user',\n",
" 'lastModifiedTimestamp': datetime.datetime(2024, 12, 3, 20, 1, 41, 745000, tzinfo=tzlocal()),\n",
" 'savedownStorageConfiguration': {'volumeName': 'SHARED_torq'},\n",
" 'azMode': 'SINGLE',\n",
" 'availabilityZoneId': 'use1-az6',\n",
" 'createdTimestamp': datetime.datetime(2024, 12, 3, 20, 1, 41, 735000, tzinfo=tzlocal()),\n",
" 'scalingGroupConfiguration': {'scalingGroupName': 'SCALING_GROUP_torq',\n",
" 'memoryReservation': 6,\n",
" 'nodeCount': 1}}"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cluster: discovery1 status is PENDING, total wait 0:00:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:00:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:01:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:01:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:02:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:02:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:03:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:03:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:04:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:04:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:05:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:05:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:06:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:06:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:07:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:07:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:08:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:08:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:09:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:09:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:10:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:10:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:11:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:11:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:12:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:12:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:13:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:13:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:14:00, waiting 30 sec ...\n",
"Cluster: discovery1 status is CREATING, total wait 0:14:30, waiting 30 sec ...\n",
"Cluster: discovery1 status is now RUNNING, total wait 0:15:00\n",
"Cluster: rdb1 status is CREATING, total wait 0:00:00, waiting 30 sec ...\n",
"Cluster: rdb1 status is CREATING, total wait 0:00:30, waiting 30 sec ...\n",
"Cluster: rdb1 status is CREATING, total wait 0:01:00, waiting 30 sec ...\n",
"Cluster: rdb1 status is now RUNNING, total wait 0:01:30\n",
"Creating: gateway1\n"
]
},
{
"data": {
"text/plain": [
"{'ResponseMetadata': {'RequestId': '15ab8f2e-8a08-469f-b0eb-ed54dcc66af7',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'application/json',\n",
" 'content-length': '1669',\n",
" 'connection': 'keep-alive',\n",
" 'date': 'Tue, 03 Dec 2024 20:18:24 GMT',\n",
" 'x-amzn-requestid': '15ab8f2e-8a08-469f-b0eb-ed54dcc66af7',\n",
" 'x-amz-apigw-id': 'CO0eAE7hoAMEtnQ=',\n",
" 'x-amzn-trace-id': 'Root=1-674f678c-22f328347fb1d17345bae243',\n",
" 'x-cache': 'Miss from cloudfront',\n",
" 'via': '1.1 2174e600dd54879ba9f49d0337eeb2dc.cloudfront.net (CloudFront)',\n",
" 'x-amz-cf-pop': 'IAD55-P1',\n",
" 'x-amz-cf-id': 'Zb-_qPIFv61cf26lMI7yc7vwNsHdIATbNENaeEg0VAQlnkajX-IHbA=='},\n",
" 'RetryAttempts': 0},\n",
" 'status': 'PENDING',\n",
" 'clusterName': 'gateway1',\n",
" 'clusterType': 'GP',\n",
" 'volumes': [{'volumeName': 'SHARED_torq', 'volumeType': 'NAS_1'}],\n",
" 'databases': [{'databaseName': 'finspace-database',\n",
" 'cacheConfigurations': [],\n",
" 'dataviewConfiguration': {'dataviewName': 'finspace-database_DBVIEW',\n",
" 'dataviewVersionId': 'jsnHC4G7CSYi9LS9fKwUNw',\n",
" 'changesetId': '2MnHBswHfdRpfkJBJVMANw',\n",
" 'segmentConfigurations': [{'dbPaths': ['/*'],\n",
" 'volumeName': 'SHARED_torq',\n",
" 'onDemand': False}]}}],\n",
" 'clusterDescription': 'Created with create_all notebook',\n",
" 'releaseLabel': '1.0',\n",
" 'vpcConfiguration': {'vpcId': 'vpc-0fe2b9c50f3ad382f',\n",
" 'securityGroupIds': ['sg-0c99f1cfb9c3c7fd9'],\n",
" 'subnetIds': ['subnet-04052219ec25b062b'],\n",
" 'ipAddressType': 'IP_V4'},\n",
" 'initializationScript': 'TorQ-Amazon-FinSpace-Starter-Pack/finspace_torq.q',\n",
" 'commandLineArguments': [{'key': 'proctype', 'value': 'gateway'},\n",
" {'key': 'procname', 'value': 'gateway1'},\n",
" {'key': 'jsonlogs', 'value': 'true'},\n",
" {'key': 'noredirect', 'value': 'true'},\n",
" {'key': 's', 'value': '2'}],\n",
" 'code': {'s3Bucket': 'kdb-demo-829845998889-kms',\n",
" 's3Key': 'code/torq_app.zip'},\n",
" 'executionRole': 'arn:aws:iam::829845998889:role/kdb-all-user',\n",
" 'lastModifiedTimestamp': datetime.datetime(2024, 12, 3, 20, 18, 24, 667000, tzinfo=tzlocal()),\n",
" 'savedownStorageConfiguration': {'volumeName': 'SHARED_torq'},\n",
" 'azMode': 'SINGLE',\n",
" 'availabilityZoneId': 'use1-az6',\n",
" 'createdTimestamp': datetime.datetime(2024, 12, 3, 20, 18, 24, 649000, tzinfo=tzlocal()),\n",
" 'scalingGroupConfiguration': {'scalingGroupName': 'SCALING_GROUP_torq',\n",
" 'memoryReservation': 6,\n",
" 'nodeCount': 1}}"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating: feed1\n"
]
},
{
"data": {
"text/plain": [
"{'ResponseMetadata': {'RequestId': 'fda47443-1942-401f-9502-dc53985704e9',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'application/json',\n",
" 'content-length': '1670',\n",
" 'connection': 'keep-alive',\n",
" 'date': 'Tue, 03 Dec 2024 20:18:27 GMT',\n",
" 'x-amzn-requestid': 'fda47443-1942-401f-9502-dc53985704e9',\n",
" 'x-amz-apigw-id': 'CO0euFWZoAMEu2w=',\n",
" 'x-amzn-trace-id': 'Root=1-674f6791-727eca3d173aa6eb1f2abc16',\n",
" 'x-cache': 'Miss from cloudfront',\n",
" 'via': '1.1 2174e600dd54879ba9f49d0337eeb2dc.cloudfront.net (CloudFront)',\n",
" 'x-amz-cf-pop': 'IAD55-P1',\n",
" 'x-amz-cf-id': 'HHSOYWC2sGWwUSCUHQOPSZuoclNp9H1W4Ap2kIFzx0RxPufEiHi7jw=='},\n",
" 'RetryAttempts': 0},\n",
" 'status': 'PENDING',\n",
" 'clusterName': 'feed1',\n",
" 'clusterType': 'GP',\n",
" 'volumes': [{'volumeName': 'SHARED_torq', 'volumeType': 'NAS_1'}],\n",
" 'databases': [{'databaseName': 'finspace-database',\n",
" 'cacheConfigurations': [],\n",
" 'dataviewConfiguration': {'dataviewName': 'finspace-database_DBVIEW',\n",
" 'dataviewVersionId': 'jsnHC4G7CSYi9LS9fKwUNw',\n",
" 'changesetId': '2MnHBswHfdRpfkJBJVMANw',\n",
" 'segmentConfigurations': [{'dbPaths': ['/*'],\n",
" 'volumeName': 'SHARED_torq',\n",
" 'onDemand': False}]}}],\n",
" 'clusterDescription': 'Created with create_all notebook',\n",
" 'releaseLabel': '1.0',\n",
" 'vpcConfiguration': {'vpcId': 'vpc-0fe2b9c50f3ad382f',\n",
" 'securityGroupIds': ['sg-0c99f1cfb9c3c7fd9'],\n",
" 'subnetIds': ['subnet-04052219ec25b062b'],\n",
" 'ipAddressType': 'IP_V4'},\n",
" 'initializationScript': 'TorQ-Amazon-FinSpace-Starter-Pack/finspace_torq.q',\n",
" 'commandLineArguments': [{'key': 'proctype', 'value': 'tradeFeed'},\n",
" {'key': 'procname', 'value': 'tradeFeed1'},\n",
" {'key': 'jsonlogs', 'value': 'true'},\n",
" {'key': 'noredirect', 'value': 'true'},\n",
" {'key': 's', 'value': '2'}],\n",
" 'code': {'s3Bucket': 'kdb-demo-829845998889-kms',\n",
" 's3Key': 'code/torq_app.zip'},\n",
" 'executionRole': 'arn:aws:iam::829845998889:role/kdb-all-user',\n",
" 'lastModifiedTimestamp': datetime.datetime(2024, 12, 3, 20, 18, 27, 845000, tzinfo=tzlocal()),\n",
" 'savedownStorageConfiguration': {'volumeName': 'SHARED_torq'},\n",
" 'azMode': 'SINGLE',\n",
" 'availabilityZoneId': 'use1-az6',\n",
" 'createdTimestamp': datetime.datetime(2024, 12, 3, 20, 18, 27, 837000, tzinfo=tzlocal()),\n",
" 'scalingGroupConfiguration': {'scalingGroupName': 'SCALING_GROUP_torq',\n",
" 'memoryReservation': 6,\n",
" 'nodeCount': 1}}"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating: monitor1\n"
]
},
{
"data": {
"text/plain": [
"{'ResponseMetadata': {'RequestId': '881e0ee9-1511-46b3-a33e-35b4d9815225',\n",
" 'HTTPStatusCode': 200,\n",
" 'HTTPHeaders': {'content-type': 'application/json',\n",
" 'content-length': '1669',\n",
" 'connection': 'keep-alive',\n",
" 'date': 'Tue, 03 Dec 2024 20:18:31 GMT',\n",
" 'x-amzn-requestid': '881e0ee9-1511-46b3-a33e-35b4d9815225',\n",
" 'x-amz-apigw-id': 'CO0fNFJLIAMER4A=',\n",
" 'x-amzn-trace-id': 'Root=1-674f6794-0d4bdb907ecf68fa7812757d',\n",
" 'x-cache': 'Miss from cloudfront',\n",
" 'via': '1.1 2174e600dd54879ba9f49d0337eeb2dc.cloudfront.net (CloudFront)',\n",
" 'x-amz-cf-pop': 'IAD55-P1',\n",
" 'x-amz-cf-id': 's2N6yEZtETnm1621l2qmuMnLRUgRZMDlajtgeP4l5RPBhwKx1i6u6w=='},\n",
" 'RetryAttempts': 0},\n",
" 'status': 'PENDING',\n",
" 'clusterName': 'monitor1',\n",
" 'clusterType': 'GP',\n",
" 'volumes': [{'volumeName': 'SHARED_torq', 'volumeType': 'NAS_1'}],\n",
" 'databases': [{'databaseName': 'finspace-database',\n",
" 'cacheConfigurations': [],\n",
" 'dataviewConfiguration': {'dataviewName': 'finspace-database_DBVIEW',\n",
" 'dataviewVersionId': 'jsnHC4G7CSYi9LS9fKwUNw',\n",
" 'changesetId': '2MnHBswHfdRpfkJBJVMANw',\n",
" 'segmentConfigurations': [{'dbPaths': ['/*'],\n",
" 'volumeName': 'SHARED_torq',\n",
" 'onDemand': False}]}}],\n",
" 'clusterDescription': 'Created with create_all notebook',\n",
" 'releaseLabel': '1.0',\n",
" 'vpcConfiguration': {'vpcId': 'vpc-0fe2b9c50f3ad382f',\n",
" 'securityGroupIds': ['sg-0c99f1cfb9c3c7fd9'],\n",
" 'subnetIds': ['subnet-04052219ec25b062b'],\n",
" 'ipAddressType': 'IP_V4'},\n",
" 'initializationScript': 'TorQ-Amazon-FinSpace-Starter-Pack/finspace_torq.q',\n",
" 'commandLineArguments': [{'key': 'proctype', 'value': 'monitor'},\n",
" {'key': 'procname', 'value': 'monitor1'},\n",
" {'key': 'jsonlogs', 'value': 'true'},\n",
" {'key': 'noredirect', 'value': 'true'},\n",
" {'key': 's', 'value': '1'}],\n",
" 'code': {'s3Bucket': 'kdb-demo-829845998889-kms',\n",
" 's3Key': 'code/torq_app.zip'},\n",
" 'executionRole': 'arn:aws:iam::829845998889:role/kdb-all-user',\n",
" 'lastModifiedTimestamp': datetime.datetime(2024, 12, 3, 20, 18, 31, 4000, tzinfo=tzlocal()),\n",
" 'savedownStorageConfiguration': {'volumeName': 'SHARED_torq'},\n",
" 'azMode': 'SINGLE',\n",
" 'availabilityZoneId': 'use1-az6',\n",
" 'createdTimestamp': datetime.datetime(2024, 12, 3, 20, 18, 30, 995000, tzinfo=tzlocal()),\n",
" 'scalingGroupConfiguration': {'scalingGroupName': 'SCALING_GROUP_torq',\n",
" 'memoryReservation': 6,\n",
" 'nodeCount': 1}}"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"for c in clusters:\n",
" # wait for a cluster?\n",
" if c['type'] == \"WAIT\":\n",
" wait_for_cluster_status(client, environmentId=ENV_ID, clusterName=c['name'], show_wait=True)\n",
" continue\n",
"\n",
" cluster_name = c['name']\n",
" cluster_type = c['type']\n",
" cluster_init = c['init']\n",
" cluster_args = c['args']\n",
"\n",
" # cluster already exists\n",
" resp = get_kx_cluster(client, environmentId=ENV_ID, clusterName=cluster_name)\n",
" if resp is not None:\n",
" print(f\"Cluster: {cluster_name} already exists\")\n",
" continue\n",
"\n",
" print(f\"Creating: {cluster_name}\")\n",
"\n",
" resp = client.create_kx_cluster(\n",
" environmentId=ENV_ID, \n",
" clusterName=cluster_name,\n",
" clusterType=cluster_type,\n",
" releaseLabel = '1.0',\n",
" executionRole=EXECUTION_ROLE,\n",
" databases=DATABASE_CONFIG,\n",
" scalingGroupConfiguration={\n",
" 'memoryReservation': 6,\n",
" 'nodeCount': 1,\n",
" 'scalingGroupName': SCALING_GROUP_NAME,\n",
" },\n",
" savedownStorageConfiguration ={ 'volumeName': VOLUME_NAME },\n",
" clusterDescription=\"Created with create_all notebook\",\n",
" code=CODE_CONFIG,\n",
" initializationScript=cluster_init,\n",
" commandLineArguments=cluster_args,\n",
" azMode=AZ_MODE,\n",
" availabilityZoneId=AZ_ID,\n",
" vpcConfiguration=VPC_CONFIG\n",
" )\n",
"\n",
" display(resp)"
]
},
{
"cell_type": "markdown",
"id": "e8491391-8c01-4190-a2bd-23fa888bf781",
"metadata": {},
"source": [
"## Wait for all clusters to finish creating"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "0946ca26-c4b0-410d-ade5-18a47bf2318a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cluster: discovery1 status is now RUNNING, total wait 0:00:00\n",
"Cluster: rdb1 status is now RUNNING, total wait 0:00:00\n",
"Cluster: hdb1 status is now RUNNING, total wait 0:00:00\n",
"Cluster: discovery1 status is now RUNNING, total wait 0:00:00\n",
"Cluster: rdb1 status is now RUNNING, total wait 0:00:00\n",
"Cluster: gateway1 status is PENDING, total wait 0:00:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:00:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:01:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:01:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:02:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:02:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:03:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:03:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:04:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:04:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:05:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:05:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:06:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:06:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:07:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:07:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:08:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:08:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:09:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:09:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:10:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:10:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:11:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:11:30, waiting 30 sec ...\n",
"Cluster: gateway1 status is CREATING, total wait 0:12:00, waiting 30 sec ...\n",
"Cluster: gateway1 status is now RUNNING, total wait 0:12:30\n",
"Cluster: feed1 status is now RUNNING, total wait 0:00:00\n",
"Cluster: monitor1 status is now RUNNING, total wait 0:00:00\n",
"** ALL DONE **\n"
]
}
],
"source": [
"# Wait for all clusters to start\n",
"for c in clusters:\n",
" cluster_name = c['name']\n",
" wait_for_cluster_status(client, environmentId=ENV_ID, clusterName=cluster_name, show_wait=True)\n",
"\n",
"print(\"** ALL DONE **\")"
]
},
{
"cell_type": "markdown",
"id": "230207e8-c297-4d7e-af65-4396fa5b4deb",
"metadata": {},
"source": [
"# List Clusters"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "1c50c578-05e8-49e7-8deb-1f6b94b10221",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>clusterName</th>\n",
" <th>status</th>\n",
" <th>clusterType</th>\n",
" <th>capacityConfiguration</th>\n",
" <th>commandLineArguments</th>\n",
" <th>clusterDescription</th>\n",
" <th>lastModifiedTimestamp</th>\n",
" <th>createdTimestamp</th>\n",
" <th>databaseName</th>\n",
" <th>cacheConfigurations</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>discovery1</td>\n",
" <td>RUNNING</td>\n",
" <td>GP</td>\n",
" <td>None</td>\n",
" <td>[{'key': 'proctype', 'value': 'discovery'}, {'key': 'procname', 'value': 'discovery1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '2'}]</td>\n",
" <td>Created with create_all notebook</td>\n",
" <td>2024-12-03 20:16:25.763000+00:00</td>\n",
" <td>2024-12-03 20:01:35.746000+00:00</td>\n",
" <td>finspace-database</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>feed1</td>\n",
" <td>RUNNING</td>\n",
" <td>GP</td>\n",
" <td>None</td>\n",
" <td>[{'key': 'proctype', 'value': 'tradeFeed'}, {'key': 'procname', 'value': 'tradeFeed1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '2'}]</td>\n",
" <td>Created with create_all notebook</td>\n",
" <td>2024-12-03 20:30:55.001000+00:00</td>\n",
" <td>2024-12-03 20:18:27.837000+00:00</td>\n",
" <td>finspace-database</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>gateway1</td>\n",
" <td>RUNNING</td>\n",
" <td>GP</td>\n",
" <td>None</td>\n",
" <td>[{'key': 'proctype', 'value': 'gateway'}, {'key': 'procname', 'value': 'gateway1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '2'}]</td>\n",
" <td>Created with create_all notebook</td>\n",
" <td>2024-12-03 20:30:55.616000+00:00</td>\n",
" <td>2024-12-03 20:18:24.649000+00:00</td>\n",
" <td>finspace-database</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>hdb1</td>\n",
" <td>RUNNING</td>\n",
" <td>GP</td>\n",
" <td>None</td>\n",
" <td>[{'key': 'proctype', 'value': 'hdb'}, {'key': 'procname', 'value': 'hdb1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '4'}]</td>\n",
" <td>Created with create_all notebook</td>\n",
" <td>2024-12-03 20:17:05.878000+00:00</td>\n",
" <td>2024-12-03 20:01:41.735000+00:00</td>\n",
" <td>finspace-database</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>monitor1</td>\n",
" <td>RUNNING</td>\n",
" <td>GP</td>\n",
" <td>None</td>\n",
" <td>[{'key': 'proctype', 'value': 'monitor'}, {'key': 'procname', 'value': 'monitor1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '1'}]</td>\n",
" <td>Created with create_all notebook</td>\n",
" <td>2024-12-03 20:29:55.787000+00:00</td>\n",
" <td>2024-12-03 20:18:30.995000+00:00</td>\n",
" <td>finspace-database</td>\n",
" <td>None</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>rdb1</td>\n",
" <td>RUNNING</td>\n",
" <td>GP</td>\n",
" <td>None</td>\n",
" <td>[{'key': 'proctype', 'value': 'rdb'}, {'key': 'procname', 'value': 'rdb1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '2'}]</td>\n",
" <td>Created with create_all notebook</td>\n",
" <td>2024-12-03 20:18:04.815000+00:00</td>\n",
" <td>2024-12-03 20:01:39.007000+00:00</td>\n",
" <td>finspace-database</td>\n",
" <td>None</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" clusterName status clusterType capacityConfiguration \\\n",
"3 discovery1 RUNNING GP None \n",
"4 feed1 RUNNING GP None \n",
"5 gateway1 RUNNING GP None \n",
"6 hdb1 RUNNING GP None \n",
"7 monitor1 RUNNING GP None \n",
"8 rdb1 RUNNING GP None \n",
"\n",
" commandLineArguments \\\n",
"3 [{'key': 'proctype', 'value': 'discovery'}, {'key': 'procname', 'value': 'discovery1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '2'}] \n",
"4 [{'key': 'proctype', 'value': 'tradeFeed'}, {'key': 'procname', 'value': 'tradeFeed1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '2'}] \n",
"5 [{'key': 'proctype', 'value': 'gateway'}, {'key': 'procname', 'value': 'gateway1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '2'}] \n",
"6 [{'key': 'proctype', 'value': 'hdb'}, {'key': 'procname', 'value': 'hdb1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '4'}] \n",
"7 [{'key': 'proctype', 'value': 'monitor'}, {'key': 'procname', 'value': 'monitor1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '1'}] \n",
"8 [{'key': 'proctype', 'value': 'rdb'}, {'key': 'procname', 'value': 'rdb1'}, {'key': 'jsonlogs', 'value': 'true'}, {'key': 'noredirect', 'value': 'true'}, {'key': 's', 'value': '2'}] \n",
"\n",
" clusterDescription lastModifiedTimestamp \\\n",
"3 Created with create_all notebook 2024-12-03 20:16:25.763000+00:00 \n",
"4 Created with create_all notebook 2024-12-03 20:30:55.001000+00:00 \n",
"5 Created with create_all notebook 2024-12-03 20:30:55.616000+00:00 \n",
"6 Created with create_all notebook 2024-12-03 20:17:05.878000+00:00 \n",
"7 Created with create_all notebook 2024-12-03 20:29:55.787000+00:00 \n",
"8 Created with create_all notebook 2024-12-03 20:18:04.815000+00:00 \n",
"\n",
" createdTimestamp databaseName cacheConfigurations \n",
"3 2024-12-03 20:01:35.746000+00:00 finspace-database None \n",
"4 2024-12-03 20:18:27.837000+00:00 finspace-database None \n",
"5 2024-12-03 20:18:24.649000+00:00 finspace-database None \n",
"6 2024-12-03 20:01:41.735000+00:00 finspace-database None \n",
"7 2024-12-03 20:18:30.995000+00:00 finspace-database None \n",
"8 2024-12-03 20:01:39.007000+00:00 finspace-database None "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"cdf = get_clusters(client, environmentId=ENV_ID)\n",
"\n",
"all_clusters = [d['name'] for d in clusters if 'name' in d]\n",
"\n",
"if cdf is not None:\n",
" cdf = cdf[cdf['clusterName'].isin(all_clusters)]\n",
"\n",
"display(cdf)"
]
},
{
"cell_type": "markdown",
"id": "9e91a23b-b100-4763-9c50-c819f5824202",
"metadata": {},
"source": [
"# All Processes Running"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "86f33240-bb12-49f3-8d9c-5783c25eb182",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Last Run: 2024-12-03 20:31:09.723819\n"
]
}
],
"source": [
"print( f\"Last Run: {datetime.datetime.now()}\" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f4f4c1f5-c164-4e93-8342-b057f66f2065",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "conda_python3",
"language": "python",
"name": "conda_python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.15"
}
},
"nbformat": 4,
"nbformat_minor": 5
}