gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb (460 lines of code) (raw):
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hlI1rYKa2IGx"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RN8N3O43QDT5"
},
"source": [
"# Vertex Prompt Optimizer Notebook UI (Preview)\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fprompts%2Fprompt_optimizer%2Fvertex_ai_prompt_optimizer_ui.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_ui.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pHyuJTFr2IGx"
},
"source": [
"# Overview\n",
"This Notebook showcases the Vertex AI prompt optimizer, a tool that iteratively optimizes prompts to suit a target model (e.g., `gemini-2.0-flash`) using target-specific metric(s).\n",
"\n",
"Key Use Cases:\n",
"\n",
"* Prompt Optimization: Enhance the quality of an initial prompt by refining its structure and content to match the target model's optimal input characteristics.\n",
"\n",
"* Prompt Translation: Adapt prompts optimized for one model to work effectively with a different target model.\n",
"\n",
"For the detailed documentation please see [here](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Getting Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tTtKHedrO1Rx"
},
"source": [
"# Step 0: Install packages and libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8-Zw72vFORz_"
},
"outputs": [],
"source": [
"!wget https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/prompts/prompt_optimizer/vapo_lib.py\n",
"import vapo_lib"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-p59jd5rOp4q"
},
"source": [
"# Step 1: Create a prompt template and system instructions\n",
"Provide your system intruction and prompt template below. Refer to [here]( https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer#template-si) for instructions.\n",
"\n",
"Prompts consist of two key components:\n",
"\n",
"- System Instruction: System instruction is the instruction that get passed to the model before any user input in the prompt. This is the fixed part of the prompt template shared across all queries for a given task.\n",
"- Prompt template: A task is the text in the prompt that you want the model to provide a response for. Context is information that you include in the prompt that the model uses or references when generating a response. These are the dynamic parts of the prompt template that changes based on the task.\n",
"\n",
"Prompt Optimizer enables the optimization or translation of the System Instruction template, while the prompt template remains essential for evaluating and selecting the best System Instruction template."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rJG1pVZO317x"
},
"outputs": [],
"source": [
"SYSTEM_INSTRUCTION = (\n",
" \"Answer the following question. Let's think step by step.\\n\" # @param\n",
")\n",
"PROMPT_TEMPLATE = \"Question: {question}\\n\\nAnswer: {target}\" # @param"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5y-cmg0TQP6v"
},
"source": [
"# Step 2: Configure project settings\n",
"To optimize the prompt for your target Google model, provide a CSV or JSONL file containing labeled validation samples (input, ground truth output pairs). Refer to [here](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer#prepare-sample-prompts) for instructions.\n",
"\n",
"Focus on examples that specifically demonstrate the issues you want to address.\n",
"Recommendation: Use 50-100 distinct samples for reliable results. However, the tool can still be effective with as few as 5 samples.\n",
"For prompt translation (e.g. 3P model to Google model, PaLM 2 to Gemini):\n",
"\n",
"Consider using the source model to label examples that the target model struggles with, helping to identify areas for improvement.\n",
"When you select a source model, you don't need to provide labels for the input examples.\n",
"While the source model selection is limited to Google models, it still supports labeled inputs from non-Google models. If you wish to select a non-Google source model, you will need to provide labels for your input examples.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mfgi_oR6tTIB"
},
"outputs": [],
"source": [
"# @markdown **Project setup**: <br/>\n",
"PROJECT_ID = \"[YOUR_PROJECT]\" # @param {type:\"string\"}\n",
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
"OUTPUT_PATH = \"[OUTPUT_PATH]\" # @param {type:\"string\"}\n",
"INPUT_DATA_PATH = \"[INPUT_DATA_PATH]\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ucebZHkHRxKH"
},
"source": [
"# Step 3: Configure optimization settings\n",
"The optimization configurations are defaulted to the values that are most commonly used, which we recommend using as the initial set-up.\n",
"\n",
"The most important settings are:\n",
"\n",
"* Target Model: Which model you are trying to optimize your prompts to.\n",
"* Optimization Mode: The mode in which you are trying to optimize your prompt with.\n",
"* Evaluation Metrics: The evaluation metrics in which you are trying to optimize your prompts against.\n",
"* Translation Source Field Name: fill in with the corresponding field name of the source text in the data if translation metrics like Comet or MetricX are selected. Otherwise, leave it as empty.\n",
"\n",
"Note that all evaluation metrics are expected to have the larger-the-better property. Therefore, we have modified the MetricX value to between 0 (worst) and 25 (best).\n",
"Refer [here](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer#configuration) to learn more about the different configuration settings and how to best utilize them."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "B2R3P8mMvK9q"
},
"outputs": [],
"source": [
"TARGET_MODEL = \"gemini-2.0-flash-001\" # @param [\"gemini-2.0-flash-lite-001\", \"gemini-2.0-flash-001\"]\n",
"OPTIMIZATION_MODE = \"instruction_and_demo\" # @param [\"instruction\", \"demonstration\", \"instruction_and_demo\"]\n",
"EVAL_METRIC = \"question_answering_correctness\" # @param [\"bleu\", \"coherence\", \"comet\", \"exact_match\", \"fluency\", \"groundedness\", \"metricx\", \"text_quality\", \"verbosity\", \"rouge_1\", \"rouge_2\", \"rouge_l\", \"rouge_l_sum\", \"safety\", \"question_answering_correctness\", \"question_answering_quality\", \"summarization_quality\", \"tool_name_match\", \"tool_parameter_key_match\", \"tool_parameter_kv_match\", \"tool_call_valid\"] {type:\"string\"}\n",
"TRANSLATION_SOURCE_FIELD_NAME = \"\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "kO7fO0qTSNLs"
},
"source": [
"# Step 4: Configure advanced optimization settings [Optional]\n",
"Refer [here](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer#configuration) to learn more about the different configuration settings and how to best utilize them."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fRHHTpaV4Xyo"
},
"outputs": [],
"source": [
"# @markdown **Instruction Optimization Configs**: <br/>\n",
"NUM_INST_OPTIMIZATION_STEPS = 10 # @param {type:\"integer\"}\n",
"\n",
"# @markdown **Demonstration Optimization Configs**: <br/>\n",
"NUM_DEMO_OPTIMIZATION_STEPS = 10 # @param {type:\"integer\"}\n",
"NUM_DEMO_PER_PROMPT = 3 # @param {type:\"integer\"}\n",
"\n",
"# @markdown **Model Configs**: <br/>\n",
"TARGET_MODEL_QPS = 3.0 # @param {type:\"number\"}\n",
"EVAL_QPS = 3.0 # @param {type:\"number\"}\n",
"\n",
"# @markdown **Multi-metric Configs**: <br/>\n",
"# @markdown Use this section only if you need more than one metric for optimization. This will override the metric you picked above.\n",
"EVAL_METRIC_1 = \"NA\" # @param [\"NA\", \"bleu\", \"coherence\", \"comet\", \"exact_match\", \"fluency\", \"groundedness\", \"metricx\", \"text_quality\", \"verbosity\", \"rouge_1\", \"rouge_2\", \"rouge_l\", \"rouge_l_sum\", \"safety\", \"question_answering_correctness\", \"question_answering_quality\", \"summarization_quality\", \"tool_name_match\", \"tool_parameter_key_match\", \"tool_parameter_kv_match\", \"tool_call_valid\"] {type:\"string\"}\n",
"EVAL_METRIC_1_WEIGHT = 0.0 # @param {type:\"number\"}\n",
"EVAL_METRIC_2 = \"NA\" # @param [\"NA\", \"bleu\", \"coherence\", \"comet\", \"exact_match\", \"fluency\", \"groundedness\", \"metricx\", \"text_quality\", \"verbosity\", \"rouge_1\", \"rouge_2\", \"rouge_l\", \"rouge_l_sum\", \"safety\", \"question_answering_correctness\", \"question_answering_quality\", \"summarization_quality\", \"tool_name_match\", \"tool_parameter_key_match\", \"tool_parameter_kv_match\", \"tool_call_valid\"] {type:\"string\"}\n",
"EVAL_METRIC_2_WEIGHT = 0.0 # @param {type:\"number\"}\n",
"EVAL_METRIC_3 = \"NA\" # @param [\"NA\", \"bleu\", \"coherence\", \"comet\", \"exact_match\", \"fluency\", \"groundedness\", \"metricx\", \"text_quality\", \"verbosity\", \"rouge_1\", \"rouge_2\", \"rouge_l\", \"rouge_l_sum\", \"safety\", \"question_answering_correctness\", \"question_answering_quality\", \"summarization_quality\", \"tool_name_match\", \"tool_parameter_key_match\", \"tool_parameter_kv_match\", \"tool_call_valid\"] {type:\"string\"}\n",
"EVAL_METRIC_3_WEIGHT = 0.0 # @param {type:\"number\"}\n",
"METRIC_AGGREGATION_TYPE = \"weighted_sum\" # @param [\"weighted_sum\", \"weighted_average\"]\n",
"\n",
"# @markdown **Misc Configs**: <br/>\n",
"PLACEHOLDER_TO_VALUE = \"{}\" # @param\n",
"RESPONSE_MIME_TYPE = \"text/plain\" # @param [\"text/plain\", \"application/json\", \"text/x.enum\"] {type:\"string\"}\n",
"RESPONSE_SCHEMA = \"\"\n",
"TARGET_LANGUAGE = \"English\" # @param [\"English\", \"French\", \"German\", \"Hebrew\", \"Hindi\", \"Italian\", \"Japanese\", \"Korean\", \"Portuguese\", \"Simplified Chinese\", \"Spanish\", \"Traditional Chinese\"] {type:\"string\"}\n",
"TOOLS = \"\" # @param\n",
"TOOL_CONFIG = \"\" # @param"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X7Mgb0EHSSFk"
},
"source": [
"# Step 5: Run Prompt Optimizer\n",
"A progress bar will appear to let you know how long the job takes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Z8NvNLTfxPTf"
},
"outputs": [],
"source": [
"import datetime\n",
"import json\n",
"import time\n",
"\n",
"timestamp = datetime.datetime.now().strftime(\"%Y-%m-%dT%H:%M:%S\")\n",
"display_name = f\"pt_{timestamp}\"\n",
"\n",
"label_enforced = vapo_lib.is_run_target_required(\n",
" [\n",
" EVAL_METRIC,\n",
" EVAL_METRIC_1,\n",
" EVAL_METRIC_2,\n",
" EVAL_METRIC_3,\n",
" ],\n",
" \"\",\n",
")\n",
"input_data_path = f\"{INPUT_DATA_PATH}\"\n",
"vapo_lib.validate_prompt_and_data(\n",
" \"\\n\".join([SYSTEM_INSTRUCTION, PROMPT_TEMPLATE]),\n",
" input_data_path,\n",
" PLACEHOLDER_TO_VALUE,\n",
" label_enforced,\n",
")\n",
"\n",
"output_path = f\"{OUTPUT_PATH}/{display_name}\"\n",
"\n",
"params = {\n",
" \"project\": PROJECT_ID,\n",
" \"num_steps\": NUM_INST_OPTIMIZATION_STEPS,\n",
" \"system_instruction\": SYSTEM_INSTRUCTION,\n",
" \"prompt_template\": PROMPT_TEMPLATE,\n",
" \"target_model\": TARGET_MODEL,\n",
" \"target_model_qps\": TARGET_MODEL_QPS,\n",
" \"target_model_location\": LOCATION,\n",
" \"eval_qps\": EVAL_QPS,\n",
" \"optimization_mode\": OPTIMIZATION_MODE,\n",
" \"num_demo_set_candidates\": NUM_DEMO_OPTIMIZATION_STEPS,\n",
" \"demo_set_size\": NUM_DEMO_PER_PROMPT,\n",
" \"aggregation_type\": METRIC_AGGREGATION_TYPE,\n",
" \"data_limit\": 50,\n",
" \"input_data_path\": input_data_path,\n",
" \"output_path\": output_path,\n",
" \"response_mime_type\": RESPONSE_MIME_TYPE,\n",
" \"response_schema\": RESPONSE_SCHEMA,\n",
" \"language\": TARGET_LANGUAGE,\n",
" \"placeholder_to_content\": json.loads(PLACEHOLDER_TO_VALUE),\n",
" \"tools\": TOOLS,\n",
" \"tool_config\": TOOL_CONFIG,\n",
" \"translation_source_field_name\": TRANSLATION_SOURCE_FIELD_NAME,\n",
"}\n",
"\n",
"if EVAL_METRIC_1 == \"NA\":\n",
" params[\"eval_metrics_types\"] = [EVAL_METRIC]\n",
" params[\"eval_metrics_weights\"] = [1.0]\n",
"else:\n",
" metrics = []\n",
" weights = []\n",
" for metric, weight in zip(\n",
" [EVAL_METRIC_1, EVAL_METRIC_2, EVAL_METRIC_3],\n",
" [EVAL_METRIC_1_WEIGHT, EVAL_METRIC_2_WEIGHT, EVAL_METRIC_3_WEIGHT],\n",
" ):\n",
" if metric == \"NA\":\n",
" break\n",
" metrics.append(metric)\n",
" weights.append(weight)\n",
" params[\"eval_metrics_types\"] = metrics\n",
" params[\"eval_metrics_weights\"] = weights\n",
"\n",
"job = vapo_lib.run_apd(params, OUTPUT_PATH, display_name)\n",
"print(f\"Job ID: {job.name}\")\n",
"\n",
"progress_form = vapo_lib.ProgressForm(params)\n",
"while progress_form.monitor_progress(job):\n",
" time.sleep(5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "lo5mcTzwSgBP"
},
"source": [
"# Step 6: Inspect the results\n",
"For a clearer look at the specific responses generated by each prompt template during the optimization process, use the cell below.\n",
"This will allow you to inspect all the predictions made by all the\n",
"generated templates during one or multiple vertex prompt optimizer runs.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1x6HSty759jY"
},
"outputs": [],
"source": [
"from IPython.display import HTML, display\n",
"\n",
"RESULT_PATH = \"[OUTPUT_PATH]\" # @param {type:\"string\"}\n",
"\n",
"results_ui = vapo_lib.ResultsUI(RESULT_PATH)\n",
"\n",
"results_df_html = \"\"\"\n",
"<style>\n",
" .scrollable {\n",
" width: 100%;\n",
" height: 80px;\n",
" overflow-y: auto;\n",
" overflow-x: hidden; /* Hide horizontal scrollbar */\n",
" }\n",
" tr:nth-child(odd) {\n",
" background: var(--colab-highlighted-surface-color);\n",
" }\n",
" tr:nth-child(even) {\n",
" background-color: var(--colab-primary-surface-color);\n",
" }\n",
" th {\n",
" background-color: var(--colab-highlighted-surface-color);\n",
" }\n",
"</style>\n",
"\"\"\"\n",
"\n",
"display(HTML(results_df_html))\n",
"display(results_ui.get_container())"
]
}
],
"metadata": {
"colab": {
"name": "vertex_ai_prompt_optimizer_ui.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}