gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb (1,281 lines of code) (raw):
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Vertex prompt optimizer Notebook SDK (Preview)\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fprompts%2Fprompt_optimizer%2Fvertex_ai_prompt_optimizer_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/prompt_optimizer/vertex_ai_prompt_optimizer_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> \n",
"\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0ccc35a93b9f"
},
"source": [
"| | | |\n",
"|-|-|-|\n",
"|Author | [Ivan Nardini](https://github.com/inardini)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## 1. Overview\n",
"\n",
"When developing Generative AI (Gen AI) applications, prompt engineering poses challenges due to its time-consuming and error-prone nature. Significant effort is involved when crafting and inputting prompts to achieve successful task completion. With the frequent release of foundational models, you face the added burden of migrating working prompts from one model version to another.\n",
"\n",
"Vertex AI prompt optimizer aims to alleviate these challenges by providing you with an intelligent prompt optimization tool. With this tool you can both translate and optimize system instruction in the prompts and the best demonstrations (examples) for prompt templates, empowering you to shape LLM responses from any source model to a target Google model.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4HKyj5KwYePX"
},
"source": [
"### Objective\n",
"\n",
"This notebook demonstrates how to leverage Vertex AI prompt optimizer to optimize a simple prompt for a Gemini model with respect to a question-answering task. The goal is to use Vertex AI prompt optimizer to find the new prompt template that generates the most accurate and grounded responses.\n",
"\n",
"This tutorial uses the following Google Cloud ML services and resources:\n",
"\n",
"- Generative AI on Vertex AI\n",
"- Vertex AI prompt optimizer\n",
"- Vertex AI Gen AI evaluation\n",
"- Vertex AI Custom job\n",
"\n",
"The steps performed include:\n",
"\n",
"1. Define the prompt template you want to optimize.\n",
"2. Prepare the prompt optimization dataset.\n",
"3. Set target model and evaluation metric.\n",
"4. Set optimization mode and steps.\n",
"5. Run the automatic prompt optimization job.\n",
"6. Collect the best prompt template and evaluation metric.\n",
"7. Validate the best prompt template."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "08d289fa873f"
},
"source": [
"### Dataset\n",
"\n",
"The dataset is a question-answering dataset generated by a simple AI cooking assistant that provides suggestions on how to prepare healthier dishes.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "aed92deeb4a0"
},
"source": [
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI\n",
"* Cloud Storage\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## 2. Before you start"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK for Python and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --quiet 'google-cloud-aiplatform[evaluation]'\n",
"%pip install --upgrade --quiet 'plotly' 'asyncio' 'tqdm' 'tenacity' 'etils' 'importlib_resources' 'fsspec' 'gcsfs' 'nbformat>=4.2.0'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e55e2195ce2d"
},
"outputs": [],
"source": [
"! mkdir -p ./tutorial/utils && wget https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/prompts/prompt_optimizer/vapo_lib.py -P ./tutorial/utils"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R5Xep4W9lq-Z"
},
"source": [
"### Restart runtime (Colab only)\n",
"\n",
"To use the newly installed packages, you must restart the runtime on Google Colab."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XRvKdaPDTznN"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" import IPython\n",
"\n",
" app = IPython.Application.instance()\n",
" app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SbmM4z7FOBpM"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
"</div>\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"Authenticate your environment on Google Colab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"# import sys\n",
"\n",
"# if \"google.colab\" in sys.modules:\n",
"# from google.colab import auth\n",
"\n",
"# auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "WReHDGG5g0XY"
},
"source": [
"#### Set your project ID and project number"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oM1iC_MfAts1"
},
"outputs": [],
"source": [
"PROJECT_ID = \"[your-project-id]\" # @param {type:\"string\"}\n",
"\n",
"# Set the project id\n",
"! gcloud config set project {PROJECT_ID}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "11a4349d673e"
},
"outputs": [],
"source": [
"PROJECT_NUMBER = !gcloud projects describe {PROJECT_ID} --format=\"get(projectNumber)\"[0]\n",
"PROJECT_NUMBER = PROJECT_NUMBER[0]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "region"
},
"source": [
"#### Region\n",
"\n",
"You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "I6FmBV2_0fBP"
},
"outputs": [],
"source": [
"REGION = \"us-central1\" # @param {type: \"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zgPO1eR3CYjk"
},
"source": [
"#### Create a Cloud Storage bucket\n",
"\n",
"Create a storage bucket to store intermediate artifacts such as datasets."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MzGDU7TWdts_"
},
"outputs": [],
"source": [
"BUCKET_NAME = \"your-bucket-name-{PROJECT_ID}-unique\" # @param {type:\"string\"}\n",
"\n",
"BUCKET_URI = f\"gs://{BUCKET_NAME}\" # @param {type:\"string\"}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NIq7R4HZCfIc"
},
"outputs": [],
"source": [
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "set_service_account"
},
"source": [
"#### Service Account and permissions\n",
"\n",
"Vertex AI Automated Prompt Design requires a service account with the following permissions:\n",
"\n",
"- `Vertex AI User` to call Vertex LLM API\n",
"- `Storage Object Admin` to read and write to your GCS bucket.\n",
"- `Artifact Registry Reader` to download the pipeline template from Artifact Registry.\n",
"\n",
"[Check out the documentation](https://cloud.google.com/iam/docs/manage-access-service-accounts#iam-view-access-sa-gcloud) to learn how to grant those permissions to a single service account."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VX9tpdtuQI5L"
},
"source": [
"> If you run following commands using Vertex AI Workbench, run directly in the terminal.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ssUJJqXJJHgC"
},
"outputs": [],
"source": [
"SERVICE_ACCOUNT = f\"{PROJECT_NUMBER}-compute@developer.gserviceaccount.com\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wqOHg5aid6HP"
},
"outputs": [],
"source": [
"for role in ['aiplatform.user', 'storage.objectAdmin', 'artifactregistry.reader']:\n",
"\n",
" ! gcloud projects add-iam-policy-binding {PROJECT_ID} \\\n",
" --member=serviceAccount:{SERVICE_ACCOUNT} \\\n",
" --role=roles/{role} --condition=None"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ek1-iTbPjzdJ"
},
"source": [
"### Set tutorial folder and workspace\n",
"\n",
"Set a local folder to collect and organize data and any tutorial artifacts."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BbfKRabXj3la"
},
"outputs": [],
"source": [
"from pathlib import Path as path\n",
"\n",
"ROOT_PATH = path.cwd()\n",
"TUTORIAL_PATH = ROOT_PATH / \"tutorial\"\n",
"TUTORIAL_PATH.mkdir(parents=True, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BaNdfftpXTIX"
},
"source": [
"Set an associated workspace to store prompt optimization results on Cloud Storage bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "joJPc3FmX1fk"
},
"outputs": [],
"source": [
"from etils import epath\n",
"\n",
"WORKSPACE_URI = epath.Path(BUCKET_URI) / \"optimization\"\n",
"INPUT_DATA_URI = epath.Path(WORKSPACE_URI) / \"data\"\n",
"\n",
"WORKSPACE_URI.mkdir(parents=True, exist_ok=True)\n",
"INPUT_DATA_URI.mkdir(parents=True, exist_ok=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "960505627ddf"
},
"source": [
"### Import libraries\n",
"\n",
"Import required libraries."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PyQmSRbKA8r-"
},
"outputs": [],
"source": [
"# Tutorial\n",
"from argparse import Namespace\n",
"import json\n",
"\n",
"# General\n",
"import logging\n",
"import warnings\n",
"\n",
"from IPython.display import HTML, display\n",
"from google.cloud import aiplatform\n",
"import pandas as pd\n",
"from sklearn.model_selection import train_test_split\n",
"from tutorial.utils import vapo_lib"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "820DIvw1o8tB"
},
"source": [
"### Libraries logging\n",
"\n",
"Configure logging for libraries to display output within the notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HKc4ZdUBo_SM"
},
"outputs": [],
"source": [
"warnings.filterwarnings(\"ignore\")\n",
"logging.getLogger(\"urllib3.connectionpool\").setLevel(logging.ERROR)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gxc7q4r-DFH4"
},
"source": [
"### Define constants\n",
"\n",
"Define some tutorial constants."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0Y5t67f3DHNm"
},
"outputs": [],
"source": [
"INPUT_DATA_FILE_URI = \"gs://github-repo/prompts/prompt_optimizer/rag_qa_dataset.jsonl\"\n",
"\n",
"EXPERIMENT_NAME = \"qa-prompt-eval\"\n",
"INPUT_OPTIMIZATION_DATA_URI = epath.Path(WORKSPACE_URI) / \"prompt_optimization_data\"\n",
"INPUT_OPTIMIZATION_DATA_FILE_URI = str(\n",
" INPUT_DATA_URI / \"prompt_optimization_dataset.jsonl\"\n",
")\n",
"OUTPUT_OPTIMIZATION_DATA_URI = epath.Path(WORKSPACE_URI) / \"optimization_jobs\"\n",
"APD_CONTAINER_URI = (\n",
" \"us-docker.pkg.dev/vertex-ai-restricted/builtin-algorithm/apd:preview_v1_0\"\n",
")\n",
"CONFIG_FILE_URI = str(WORKSPACE_URI / \"config\" / \"config.json\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "init_aip:mbsdk,all"
},
"source": [
"### Initialize Vertex AI SDK for Python\n",
"\n",
"Initialize the Vertex AI SDK for Python for your project."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bQMc2Uwf0fBQ"
},
"outputs": [],
"source": [
"aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EdvJRUWRNGHE"
},
"source": [
"## 3. Automated prompt design with Vertex AI prompt optimizer"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "mmTotjRAJplw"
},
"source": [
"### Load the dataset\n",
"\n",
"Load the cooking question-answer dataset from a Google Cloud Storage bucket. The dataset contains the following columns:\n",
"\n",
"* `user_question`: The cooking question posed by the user to the AI cooking assistant.\n",
"* `context`: Relevant information retrieved to answer the user's question.\n",
"* `prompt`: The content fed to the language model to generate an answer.\n",
"* `answer`: The generated answer from the language model.\n",
"* `reference`: The ground truth answer—the ideal response the user expects from the AI cooking assistant."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LA7aG08wJtVm"
},
"outputs": [],
"source": [
"prompt_optimization_df = pd.read_json(INPUT_DATA_FILE_URI, lines=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1xn-pz3v5HVK"
},
"outputs": [],
"source": [
"prompt_optimization_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E2l7JwuDcPqX"
},
"source": [
"Print an example of the cooking question-answer dataset. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PsXdJBJXiaVH"
},
"outputs": [],
"source": [
"vapo_lib.print_df_rows(prompt_optimization_df, n=1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "E5SmBApC-WDg"
},
"source": [
"### Evaluate the system instruction in the original prompt template\n",
"\n",
"Assess the original prompt's effectiveness for our AI cooking assistant's question-answering task using Vertex AI's Gen AI Evaluation service. This service offers various metrics and methods to evaluate generative models, which enables comparing the model's performance against our own expectations and criteria.\n",
"\n",
"Specifically, you focus on the quality and groundedness of the answers generated in response to the prompt using a test dataset.\n",
"\n",
"To learn more, see [Gen AI evaluation service overview](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview). \n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KhWt4iaGRBdl"
},
"outputs": [],
"source": [
"train_prompt_optimization_df, test_prompt_optimization_df = train_test_split(\n",
" prompt_optimization_df, test_size=0.8, random_state=8\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qA-dl76E-H23"
},
"outputs": [],
"source": [
"evaluation_qa_results = [\n",
" (\n",
" \"qa_eval_result_without_prompt_optimization\",\n",
" vapo_lib.evaluate_task(\n",
" df=test_prompt_optimization_df,\n",
" prompt_col=\"prompt\",\n",
" reference_col=\"reference\",\n",
" response_col=\"answer\",\n",
" experiment_name=EXPERIMENT_NAME,\n",
" eval_metrics=[\"question_answering_quality\", \"groundedness\"],\n",
" eval_sample_n=len(test_prompt_optimization_df),\n",
" ),\n",
" )\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_9ZMmVHZfl5O"
},
"source": [
"Plot the evaluation metrics."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yTZKlgOk-0qz"
},
"outputs": [],
"source": [
"vapo_lib.plot_eval_metrics(evaluation_qa_results)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Rp1n1aMACzSW"
},
"source": [
"### Optimize the prompt template with Vertex AI prompt optimizer\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "h1650lf3X8xW"
},
"source": [
"#### Prepare the prompt template you want to optimize\n",
"\n",
"A prompt consists of two key parts:\n",
"\n",
"* **System Instruction Template** which is a fixed part of the prompt that control or alter the model's behavior across all queries for a given task.\n",
"\n",
"* **Prompt Template** which is a dynamic part of the prompt that changes based on the task. Prompt template includes context, task and more. To learn more, see [components of a prompt](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-design-strategies#components-of-a-prompt) in the official documentation.\n",
"\n",
"In this scenario, you use Vertex AI prompt optimizer to optimize a simple system instruction template. And you use some examples in the remaining prompt template for evaluating different instruction templates along the optimization process."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e90a4e8c3bb6"
},
"source": [
"> Having the `target` placeholder in the prompt template is optional. It represents the prompt's ground truth response in your prompt optimization dataset that you aim to optimize for your templates. If you don't have the prompt's ground truth response, remember to set the `source_model` parameter to your prompt optimizer configuration (see below) instead of adding ground truth responses. Vertex AI prompt optimizer would run your sample prompts on the source model to generate the ground truth responses for you.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "oberF30r1ZgF"
},
"outputs": [],
"source": [
"SYSTEM_INSTRUCTION_TEMPLATE = \"\"\"\n",
"Given a question with context, provide the correct answer to the question.\n",
"\"\"\"\n",
"\n",
"PROMPT_TEMPLATE = \"\"\"\n",
"Some examples of correct answer to a question are:\n",
"Question: {question}\n",
"Context: {ctx}\n",
"Answer: {target}\n",
"\"\"\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a1TCgXsrXztm"
},
"source": [
"#### Prepare the prompt optimization dataset\n",
"\n",
"To use Vertex AI prompt optimizer, you'll need a CSV or JSONL file with labeled examples. These examples should follow a specific naming convention. For details see [Optimize prompts](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer).\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7nCeas2eEirf"
},
"source": [
"> For effective **prompt optimization**, provide a dataset of examples where your model is poor in performance when using current system instruction template. For reliable results, use 50-100 distinct samples.\n",
"\n",
"> In case of **prompt migration**, consider using the source model to label examples that the target model struggles with, helping to identify areas for improvement.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Yd2YDw5q1cAT"
},
"outputs": [],
"source": [
"prepared_train_prompt_optimization_df = train_prompt_optimization_df.copy()\n",
"\n",
"# Prepare optimization dataset columns\n",
"prepared_train_prompt_optimization_df = prepared_train_prompt_optimization_df.rename(\n",
" columns={\"user_question\": \"question\", \"context\": \"ctx\", \"reference\": \"target\"}\n",
")\n",
"\n",
"# Remove uneccessary columns\n",
"prepared_train_prompt_optimization_df = prepared_train_prompt_optimization_df.drop(\n",
" columns=[\"prompt\", \"answer\"]\n",
")\n",
"\n",
"# Reorder columns\n",
"prepared_train_prompt_optimization_df = prepared_train_prompt_optimization_df[\n",
" [\"question\", \"ctx\", \"target\"]\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZbTqB2o5ZslT"
},
"source": [
"Print some examples of the prompt optimization dataset. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_DUFEAb82eEi"
},
"outputs": [],
"source": [
"prepared_train_prompt_optimization_df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nF3XY_d_yB-K"
},
"source": [
"#### Upload samples to bucket\n",
"\n",
"Once you prepare your prompt optimization dataset, you can upload them on Cloud Storage bucket."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "155paLgGUXOm"
},
"outputs": [],
"source": [
"prepared_train_prompt_optimization_df.to_json(\n",
" INPUT_OPTIMIZATION_DATA_FILE_URI, orient=\"records\", lines=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F5RD0l2xX-FI"
},
"source": [
"#### Configure optimization settings\n",
"\n",
"Vertex AI prompt optimizer lets you control the optimization process by specifying what to optimize (instructions only, demonstrations only, or both), providing a system instruction and prompt template, and selecting the target model. You can optionally refine the optimization with some advanced settings like its duration and the number of optimization iterations it runs, which models the Vertex AI prompt optimizer uses, and other parameters to control the structure and content of prompts.\n",
"\n",
"Below are some common and recommended default configurations. For more advanced control, you can learn and explore more about all the parameters and how to best use them in the [detailed documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer).\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "sFHutXhgeqRx"
},
"outputs": [],
"source": [
"PROMPT_OPTIMIZATION_JOB = \"auto-prompt-design-job-\" + vapo_lib.get_id()\n",
"OUTPUT_OPTIMIZATION_RUN_URI = str(\n",
" OUTPUT_OPTIMIZATION_DATA_URI / PROMPT_OPTIMIZATION_JOB\n",
")\n",
"\n",
"args = Namespace(\n",
" # Basic configuration\n",
" system_instruction=SYSTEM_INSTRUCTION_TEMPLATE, # System instructions for the target model. String.\n",
" prompt_template=PROMPT_TEMPLATE, # Template for prompts, String.\n",
" target_model=\"gemini-2.0-flash-001\", # Target model for optimization. String. Supported models: \"gemini-2.0-flash-lite-001\", \"gemini-2.0-flash-001\"\n",
" optimization_mode=\"instruction\", # Optimization mode. String. Supported modes: \"instruction\", \"demonstration\", \"instruction_and_demo\"\n",
" eval_metrics_types=[\n",
" \"question_answering_correctness\",\n",
" \"groundedness\",\n",
" ], # List of evaluation metrics. List of strings. Supported metrics: \"bleu\", \"coherence\", \"comet\", \"exact_match\", \"fluency\", \"groundedness\", \"metricx\", \"rouge_1\", \"rouge_2\", \"rouge_l\", \"rouge_l_sum\", \"safety\", \"question_answering_correctness\", \"question_answering_quality\", \"summarization_quality\", \"text_quality\", \"verbosity\", \"tool_call_valid\", \"tool_name_match\", \"tool_parameter_key_match\", \"tool_parameter_kv_match\"\n",
" eval_metrics_weights=[\n",
" 0.9,\n",
" 0.1,\n",
" ], # Weights for evaluation metrics. List of floats. Length must match eval_metrics_types. Should sum to 1.\n",
" aggregation_type=\"weighted_sum\", # Aggregation type for evaluation metrics. String. Supported aggregation types: \"weighted_sum\", \"weighted_average\"\n",
" input_data_path=INPUT_OPTIMIZATION_DATA_FILE_URI, # Cloud Storage URI to input optimization data. String.\n",
" output_path=OUTPUT_OPTIMIZATION_RUN_URI, # Cloud Storage URI to save optimization results. String.\n",
" project=PROJECT_ID, # Google Cloud project ID. String.\n",
" # (Optional) Advanced configuration\n",
" num_steps=10, # Number of iterations in instruction optimization mode. Integer between 10 and 20.\n",
" num_demo_set_candidates=10, # Number of demonstrations evaluated in instruction and instruction_and_demo mode. Integer between 10 and 30.\n",
" demo_set_size=3, # Number of demonstrations generated per prompt. Integer between 3 and 6.\n",
" target_model_location=\"us-central1\", # Location of the target model. String. Default us-central1.\n",
" source_model=\"\", # Google model that the system instructions and prompts were previously used with. String. Not needed if you provide target column.\n",
" source_model_location=\"\", # Location of the source model. String. Default us-central1. Not needed if you provide target column.\n",
" target_model_qps=1, # The queries per second (QPS) sent to the target model. Integer greater or equal than 1 depending on your quota.\n",
" optimizer_model_qps=1, # The queries per second (QPS) sent to the optimization model. Integer greater or equal than 1 depending on your quota.\n",
" eval_qps=1, # The queries per second (QPS) sent to the eval model. Integer greater or equal than 1 depending on your quota.\n",
" source_model_qps=\"\", # The queries per second (QPS) sent to the source model. Integer greater or equal than 1 depending on your quota.\n",
" response_mime_type=\"text/plain\", # MIME response type that the target model uses. String. Supported response: text/plain, text/x.enum, application/json.\n",
" response_schema=\"\", # The Vertex AI's Controlled Generation response schema that the target model uses to generate answers. String.\n",
" language=\"English\", # Language of the system instructions. String. Supported languages: \"English\", \"French\", \"German\", \"Hebrew\", \"Hindi\", \"Italian\", \"Japanese\", \"Korean\", \"Portuguese\", \"Simplified Chinese\", \"Spanish\", \"Traditional Chinese\"\n",
" placeholder_to_content=json.loads(\n",
" \"{}\"\n",
" ), # Placeholder to replace any parameter in the system instruction. Dict.\n",
" data_limit=10, # Amount of data used for validation. Integer between 5 and 100.\n",
" translation_source_field_name=\"\", # Fill in with the corresponding field name of the source text in the data if translation metrics like Comet or MetricX are selected. Otherwise, leave it as empty.\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Jd_uzQYQx6L7"
},
"source": [
"#### Upload Vertex AI prompt optimizer config to Cloud Storage\n",
"\n",
"After define the Vertex AI prompt optimizer configuration, upload them on Cloud Storage bucket.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "iqiv8ApR_SAM"
},
"outputs": [],
"source": [
"args = vars(args)\n",
"\n",
"with epath.Path(CONFIG_FILE_URI).open(\"w\") as config_file:\n",
" json.dump(args, config_file)\n",
"config_file.close()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "spqgBT8hYAle"
},
"source": [
"#### Run the automatic prompt optimization job\n",
"\n",
"Now you are ready to run your first Vertex AI prompt optimizer job using the Vertex AI SDK for Python.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "nmuRfNlCHuiw"
},
"source": [
"> This prompt optimization job requires ~ 40 minutes to run.\n",
"\n",
"> Be sure you have provisioned enough queries per minute (QPM) quota implementing the recommended QPM for each model. If you configure the Vertex AI prompt optimizer with a QPM that is higher than the QPM than you have access to, the job might fail. [Check out](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/prompt-optimizer#before-you-begin) the documentation to know more.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GtPnvKIpUQ3q"
},
"outputs": [],
"source": [
"WORKER_POOL_SPECS = [\n",
" {\n",
" \"machine_spec\": {\n",
" \"machine_type\": \"n1-standard-4\",\n",
" },\n",
" \"replica_count\": 1,\n",
" \"container_spec\": {\n",
" \"image_uri\": APD_CONTAINER_URI,\n",
" \"args\": [\"--config=\" + CONFIG_FILE_URI],\n",
" },\n",
" }\n",
"]\n",
"\n",
"custom_job = aiplatform.CustomJob(\n",
" display_name=PROMPT_OPTIMIZATION_JOB,\n",
" worker_pool_specs=WORKER_POOL_SPECS,\n",
")\n",
"\n",
"custom_job.run(service_account=SERVICE_ACCOUNT, sync=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3YwwKBhtJ4ut"
},
"source": [
"### Collect and display the optimization results\n",
"\n",
"Vertex AI prompt optimizer returns both optimized templates and evaluation results for either instruction, or demostrations, or both depending on the optimization mode you define as JSONL files on Cloud Storage bucket. Those results help you understand the optimization process.\n",
"\n",
"In this case, you want to collect the optimized templates and evaluation results for the system instruction.\n",
"\n",
"Below you use a helper function to display those results."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-6ohF0LJIhvU"
},
"outputs": [],
"source": [
"results_ui = vapo_lib.ResultsUI(OUTPUT_OPTIMIZATION_RUN_URI)\n",
"results_df_html = \"\"\"\n",
"\n",
"\"\"\"\n",
"\n",
"display(HTML(results_df_html))\n",
"display(results_ui.get_container())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TrMrbcA5Gzep"
},
"source": [
"### Evaluate the new prompt template with the optimized instruction."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bGRELw3U3I28"
},
"source": [
"#### Generate new responses using the optimized system instruction.\n",
"\n",
"Set the optimized system instruction template you get from Vertex AI prompt optimizer job."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "BVamCk10KYHu"
},
"outputs": [],
"source": [
"OPTIMIZED_SYSTEM_INSTRUCTION_TEMPLATE = \"You are a culinary expert. Use the provided cooking tips and your culinary expertise to answer the following question in a way that is comprehensive, engaging, and easy for a home cook to understand.\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1U79bZ_KTM6T"
},
"source": [
"Prepare optimized prompts using the optimized system instruction template."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GXDU_ydAG5ak"
},
"outputs": [],
"source": [
"OPTIMIZED_PROMPT_TEMPLATE = (\n",
" OPTIMIZED_SYSTEM_INSTRUCTION_TEMPLATE\n",
" + \"\\nQuestion: \\n{question}\"\n",
" + \"\\nContext: \\n{context}\"\n",
" + \"\\nAnswer:\"\n",
")\n",
"\n",
"optimized_prompts = [\n",
" OPTIMIZED_PROMPT_TEMPLATE.format(question=q, context=c)\n",
" for q, c in zip(\n",
" test_prompt_optimization_df[\"user_question\"].to_list(),\n",
" test_prompt_optimization_df[\"context\"].to_list(),\n",
" )\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sofPH74kT165"
},
"source": [
"Leverage Gemini API on Vertex AI to send parallel generation requests."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qG6QJW8alttS"
},
"outputs": [],
"source": [
"gemini_llm = vapo_lib.init_new_model(model_name=\"gemini-2.0-flash\")\n",
"\n",
"gemini_predictions = [\n",
" vapo_lib.async_generate(p, model=gemini_llm) for p in optimized_prompts\n",
"]\n",
"\n",
"gemini_predictions_col = await tqdm_asyncio.gather(*gemini_predictions)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SEVi8m4gUMqP"
},
"source": [
"Prepare the test data and visualize the resulting dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8khmj0oATV7O"
},
"outputs": [],
"source": [
"test_prompt_optimization_df[\"optimized_prompt_with_vapo\"] = optimized_prompts\n",
"test_prompt_optimization_df[\"gemini_answer_with_vapo\"] = gemini_predictions_col"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0s2pAKq4TaiJ"
},
"outputs": [],
"source": [
"vapo_lib.print_df_rows(test_prompt_optimization_df, n=1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D1wxiPhv21TT"
},
"source": [
"#### Evaluate new responses using Vertex AI Gen AI evaluation\n",
"\n",
"Use the generated responses with the optimized prompt to run a new round of evaluation with Vertex AI Gen AI Evaluation.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5Ebtvk0fKApV"
},
"outputs": [],
"source": [
"evaluation_qa_results.append(\n",
" (\n",
" \"qa_eval_result_new_model_with_prompt_optimization\",\n",
" vapo_lib.evaluate_task(\n",
" df=test_prompt_optimization_df,\n",
" prompt_col=\"optimized_prompt_with_vapo\",\n",
" reference_col=\"reference\",\n",
" response_col=\"gemini_answer_with_vapo\",\n",
" experiment_name=EXPERIMENT_NAME,\n",
" eval_metrics=[\"question_answering_quality\", \"groundedness\"],\n",
" eval_sample_n=len(test_prompt_optimization_df),\n",
" ),\n",
" )\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BkjdOy0IU1Y2"
},
"source": [
"Inspect evaluation results."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wJXNAnJjmnga"
},
"outputs": [],
"source": [
"vapo_lib.plot_eval_metrics(evaluation_qa_results)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## 4. Clean up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WRY_3wh1GVNm"
},
"outputs": [],
"source": [
"delete_bucket = False\n",
"delete_job = False\n",
"delete_experiment = False\n",
"delete_tutorial = False\n",
"\n",
"if delete_bucket:\n",
" ! gsutil rm -r $BUCKET_URI\n",
"\n",
"if delete_job:\n",
" custom_job.delete()\n",
"\n",
"if delete_experiment:\n",
" experiment = aiplatform.Experiment(experiment_name=EXPERIMENT_NAME)\n",
" experiment.delete()\n",
"\n",
"if delete_tutorial:\n",
" import shutil\n",
"\n",
" shutil.rmtree(str(TUTORIAL_PATH))"
]
}
],
"metadata": {
"colab": {
"name": "vertex_ai_prompt_optimizer_sdk.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}