open-models/get_started_with_model_garden_sdk.ipynb (877 lines of code) (raw):
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ur8xi4C7S06n"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JAPoU8Sm5E6e"
},
"source": [
"# Get started with Vertex AI Model Garden SDK\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/tree/main/open-models/get_started_with_model_garden_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fget_started_with_model_garden_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/get_started_with_model_garden_sdk.ipynb\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/tree/main/open-models/get_started_with_model_garden_sdk.ipynb\">\n",
" <img width=\"32px\" src=\"https://www.svgrepo.com/download/217753/github.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/get_started_with_model_garden_sdk.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "84f0f73a0f76"
},
"source": [
"| Author |\n",
"| --- |\n",
"| [Ivan Nardini](https://github.com/inardini) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tvgnzT1CKxrO"
},
"source": [
"## Overview\n",
"\n",
"Have you ever tried to deploy open models on Vertex AI? Deploying open models on Vertex AI can sometimes involve navigating numerous API calls and parameters, which can lead to a slower, more error-prone experience.\n",
"\n",
"But what if deploying state-of-the-art open models would be as simple as choosing your model and hitting the 'deploy' button?\n",
"\n",
"The new Vertex AI Model Garden CLI and SDK, powered by the new Deploy API, are designed to do just that. These tools are truly model-centric, providing you with a more consistent, and fluid experience. The SDK simplifies prototyping, removing the need for users to specify container details. The CLI provides a precise and interactive command-line interface for managing models, offering a programmatic alternative to the UI's one-click deployment and enabling scriptable automation.\n",
"\n",
"This tutorial shows how to use the Vertex AI Model Garden SDK for deploying your open models on Vertex AI.\n",
"\n",
"You will learn how to:\n",
"\n",
"- Find the models that you can deploy\n",
"- Deploy your 1st Model Garden model\n",
"- Handle with some advanced usage including setting deployment parameters and error handling\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "61RBz8LLbxCR"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YGcCmlbabAZZ"
},
"source": [
"orgpolicy.googleapis.com"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK and other required packages\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --upgrade --force-reinstall --quiet 'google-cloud-aiplatform>=1.84.0' 'openai' 'google-auth' 'requests'"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DF4l8DTdWgPY"
},
"source": [
"### Set Google Cloud project information\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nqwi-5ufWp_B"
},
"outputs": [],
"source": [
"# Use the environment variable if the user doesn't provide Project ID.\n",
"import os\n",
"\n",
"import vertexai\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "haaUJ5VaWajl"
},
"source": [
"## Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kCp1oCOyWfZe"
},
"outputs": [],
"source": [
"import base64\n",
"import io\n",
"import os\n",
"\n",
"from PIL import Image\n",
"import google.auth\n",
"import matplotlib.pyplot as plt\n",
"import openai\n",
"from vertexai.preview import model_garden"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fvVKEpROoXW5"
},
"source": [
"## Define helpers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9AoCHkCNoaLG"
},
"outputs": [],
"source": [
"def print_models(data_list: list[str], items_per_line: int = 2) -> None:\n",
" \"\"\"Prints the list with a specified number of items per line with index and emojis,\n",
" and includes the total count.\"\"\"\n",
" print(\"🌟--- Models available ---🌟\")\n",
" print(\"\\n\")\n",
" print(f\"🔢 Total models: {len(data_list)} 🔢\\n\") # Print the count here\n",
"\n",
" for i, item in enumerate(data_list):\n",
" print(f\"✨ {item} \", end=\"\")\n",
" if (i + 1) % items_per_line == 0:\n",
" print()\n",
" else:\n",
" print(\" --- \", end=\"\")\n",
"\n",
" if len(data_list) % items_per_line != 0:\n",
" print()\n",
"\n",
"\n",
"def plot_image_from_bytes(image_bytes_string: str) -> None:\n",
" \"\"\"Plots an image from a base64-encoded bytes string.\"\"\"\n",
" # Decode the base64 string\n",
" image_bytes = base64.b64decode(image_bytes_string)\n",
"\n",
" # Create a BytesIO object\n",
" image_stream = io.BytesIO(image_bytes)\n",
"\n",
" # Open the image using Pillow library\n",
" image = Image.open(image_stream)\n",
"\n",
" # Plot the image using matplotlib\n",
" plt.imshow(image)\n",
" plt.axis(\"off\")\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dJmOAajdW7qQ"
},
"source": [
"## Find the models that you can deploy\n",
"\n",
"In Vertex AI Model Garden, you can discover and deploy a wide range of open-source models.\n",
"\n",
"Many of these models are directly supported in Vertex AI Model Garden with some pre-configured for optimized deployment on Vertex AI. When the open model is not available in Vertex AI Model Garden or you want to deploy your model from HF hub, you can leverage the Hugging Face gallery which gives you access to more that 1M models.\n",
"\n",
"With Vertex AI Model Garden SDK, you can models that you can deploy and record the model ID to deploy. You can optionally list the supported Hugging Face models in Model Garden and even filter them by model names.\n",
"\n",
"Let's check with Gemma models are available in Vertex AI Model Garden."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2l5LsbqTevUr"
},
"outputs": [],
"source": [
"model_garden_models = model_garden.list_deployable_models(\n",
" model_filter=\"gemma\", list_hf_models=False\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "z5Fa1spZr2YZ"
},
"outputs": [],
"source": [
"print_models(model_garden_models, items_per_line=3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EWRuFtX4iXW4"
},
"source": [
"To include Gemma models that are available via Hugging Face Gallery, you can enable `list_hf_models` flag."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e9zjeZ-rrPPk"
},
"outputs": [],
"source": [
"deployable_models = model_garden.list_deployable_models(\n",
" model_filter=\"gemma\", list_hf_models=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "On3fzp-erTse"
},
"outputs": [],
"source": [
"print_models(deployable_models)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3BOPydegEkP2"
},
"source": [
"## Deploy your 1st Model Garden model\n",
"\n",
"To deploy an open model, Vertex AI Model Garden SDK provides the `OpenModel` which simplifies the process of deploying these models to Vertex AI for inference.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4qGzyKIPrkFE"
},
"outputs": [],
"source": [
"model_id = \"google/gemma3@gemma-3-1b-it\"\n",
"\n",
"gemma_model = model_garden.OpenModel(model_id)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vSFfpZ7Nxlkt"
},
"source": [
"### Check the deployment configuration"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cA_yIRK-t_2T"
},
"source": [
"After you initiate the model, use `list_deploy_options()` method to discover the verified deployment configurations supported by a specific model.\n",
"\n",
"This is important to verify if you have enough resources to deploy the model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PaOvRoFZt9T1"
},
"outputs": [],
"source": [
"gemma_model.list_deploy_options()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C3OKT16hugDY"
},
"source": [
"### Deploy the model\n",
"\n",
"Now that you know how the model will be deployed, let's use the `deploy()` method to serve the selected open model to a Vertex AI Endpoint. Depending on the model, the deployment would require some minutes.\n",
"\n",
"> **Note**: If the model has an End User License Agreement (EULA), you can accept it using `accept_eula` flag.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0YqJ9pKi0cjz"
},
"outputs": [],
"source": [
"gemma_endpoint = gemma_model.deploy(accept_eula=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "tmuwCPShxso1"
},
"source": [
"### Generate predictions\n",
"\n",
"After the model gets deployed, you can use the Vertex AI SDK."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vVHOFm7g_Lud"
},
"outputs": [],
"source": [
"prediction = gemma_endpoint.predict(\n",
" instances=[{\"prompt\": \"Tell me a joke\", \"temperature\": 0.7, \"max_tokens\": 50}]\n",
")\n",
"print(prediction.predictions[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xtsK74K7_K3j"
},
"source": [
"Additionally, you can get your prediction using ChatCompletion API from the OpenAI SDK."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "acGaSDaH9YAp"
},
"outputs": [],
"source": [
"creds, project = google.auth.default()\n",
"auth_req = google.auth.transport.requests.Request()\n",
"creds.refresh(auth_req)\n",
"\n",
"gemma_endpoint_url = f\"https://{LOCATION}-aiplatform.googleapis.com/v1beta1/{gemma_endpoint.resource_name}\"\n",
"\n",
"client = openai.OpenAI(base_url=gemma_endpoint_url, api_key=creds.token)\n",
"\n",
"prediction = client.chat.completions.create(\n",
" model=\"\",\n",
" messages=[{\"role\": \"user\", \"content\": \"Tell me a joke\"}],\n",
" temperature=0.7,\n",
" max_tokens=50,\n",
")\n",
"\n",
"print(prediction.choices[0].message.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sai-h7DryWKi"
},
"source": [
"## Advanced configuration and error handling\n",
"\n",
"The Vertex AI Model Garden SDK provides a customizable and reliable interface to deploy your open models.\n",
"\n",
"You can specify advanced deployment configurations. And it provides a solid error handling to ensure robust and stable deployments, providing informative messages when problems arise.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KRAuXc76v4pY"
},
"source": [
"### Specify optional deployement parameters\n",
"\n",
"With the `deploy()` method, you can configure various deployment settings, including:\n",
"\n",
"- Compute Resources: Machine type, number of replicas (min/max), accelerator type and count.\n",
"\n",
"- Infrastructure: Spot VMs, reservation affinity, dedicated endpoints.\n",
"\n",
"- Serving Container: Specify custom container images, ports, health checks, and environment variables for advanced customization.\n",
"\n",
"Let's deploy a diffusion model with some additional deployment parameters."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Pf3egfPPv_E6"
},
"outputs": [],
"source": [
"sd_model = model_garden.OpenModel(\"stabilityai/stable-diffusion-xl-base-1.0\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "-c6-ocHPKca0"
},
"source": [
"You deploy the model to a Vertex AI Endpoint."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "GCnpQHQZwBFk"
},
"outputs": [],
"source": [
"sd_endpoint = sd_model.deploy(\n",
" machine_type=\"g2-standard-4\",\n",
" accelerator_type=\"NVIDIA_L4\",\n",
" accelerator_count=1,\n",
" min_replica_count=1,\n",
" max_replica_count=1,\n",
" endpoint_display_name=\"sd-endpoint\",\n",
" model_display_name=\"sd-model\",\n",
" deploy_request_timeout=3 * 60 * 60,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fS1FFbbPWiLN"
},
"source": [
"Generate your image."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VjoOS7irWqx0"
},
"outputs": [],
"source": [
"prediction = sd_endpoint.predict(instances=[\"A cat\"])\n",
"plot_image_from_bytes(prediction.predictions[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "p6_t8l_R07QJ"
},
"source": [
"### Error Handling"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vdmv4qVOVVrY"
},
"source": [
"#### Model not found\n",
"\n",
"You're trying to deploy a model named `google/some-model@some-version`. This string represents the model's identifier and version within the Model Garden (or potentially another registry). However, Vertex AI cannot find a model matching that identifier and version.\n",
"\n",
"The `model.deploy()` call will raise a Python exception. This is crucial because it immediately signals that something went wrong."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dNwuhhO61DvW"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\"google/some-model@some-version\")\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Xl8deqpvVyH1"
},
"source": [
"#### Malformed model name\n",
"\n",
"You're attempting to deploy a model using the identifier `publisher/google/some-model@some-version`. The problem here isn't that the model doesn't exist, but rather that the format of the model name itself is incorrect according to the Vertex AI Model Garden's expected structure.\n",
"\n",
"Similar to the \"Model Not Found\" case, `model.deploy()` will raise a Python exception. This time, it will be an exception indicating an invalid input."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wtgkO-Rd15kR"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\"publisher/google/some-model@some-version\")\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "O2uSTNqpWYRP"
},
"source": [
"#### Insufficient quota\n",
"\n",
"You are trying to deploy the model `meta/llama3_1@llama-3.1-8b-instruct`. This time, the model name is correctly formatted, and the model exists. However, your Google Cloud project has insufficient quota to deploy this specific model.\n",
"\n",
"> Quotas are limits set by Google Cloud to control resource usage and prevent accidental overspending. Relevant quotas could include Compute Engine Quotas which are limits on the number of CPUs, GPUs, or the amount of RAM you can use in a given region. As well as Vertex AI Quota which are specific limits on the number of model deployments, prediction requests, or other Vertex AI-specific operations.\n",
"\n",
"As with the previous cases, the `model.deploy()` call will raise a Python exception. This indicates that the deployment failed due to a quota issue."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "L0iTgub52Osg"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\n",
" \"publishers/deepseek-ai/models/deepseek-r1@deepseek-r1\"\n",
" )\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "S-_Y3_SrXWV1"
},
"source": [
"#### Organization policy\n",
"\n",
"You're attempting to deploy the model `stabilityai/stable-diffusion-xl-base-1.0`. The model name is correct, the model exists, and you have sufficient quota. However, an organizational policy prevents you from deploying this specific model.\n",
"\n",
"> Organizational policies are rules set by your Google Cloud organization's administrator to control which resources and services can be used within the organization. These policies are often used for security, compliance, and cost control.\n",
"\n",
"The `model.deploy()` call will raise a Python exception like the one below:\n",
"```\n",
"---------------------------------------------------------------------------\n",
"_InactiveRpcError Traceback (most recent call last)\n",
"/usr/local/lib/python3.11/dist-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)\n",
" 75 try:\n",
"---> 76 return callable_(*args, **kwargs)\n",
" 77 except grpc.RpcError as exc:\n",
"\n",
"6 frames\n",
"_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:\n",
"\tstatus = StatusCode.FAILED_PRECONDITION\n",
"\tdetails = \"Organization Policy constraint `constraints/vertexai.allowedModels` violated for `projects/your-project-id` attempting to use a disallowed Gen AI model stable-diffusion-xl-base-1.0. Please contact your organization administrator to fix this violation. For more info, see https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access.\"\n",
"\tdebug_error_string = \"UNKNOWN:Error received from peer ipv4:your-ipv4 {created_time:\"2025-01-16T20:52:11.244017603+00:00\", grpc_status:9, grpc_message:\"Organization Policy constraint `constraints/vertexai.allowedModels` violated for `projects/your-project-id` attempting to use a disallowed Gen AI model stable-diffusion-xl-base-1.0. Please contact your organization administrator to fix this violation. For more info, see https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access.\"}\"\n",
">\n",
"\n",
"The above exception was the direct cause of the following exception:\n",
"\n",
"FailedPrecondition Traceback (most recent call last)\n",
"/usr/local/lib/python3.11/dist-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)\n",
" 76 return callable_(*args, **kwargs)\n",
" 77 except grpc.RpcError as exc:\n",
"---> 78 raise exceptions.from_grpc_error(exc) from exc\n",
" 79\n",
" 80 return error_remapped_callable\n",
"\n",
"FailedPrecondition: 400 Organization Policy constraint `constraints/vertexai.allowedModels` violated for `projects/your-project-id` attempting to use a disallowed Gen AI model stable-diffusion-xl-base-1.0. Please contact your organization administrator to fix this violation. For more info, see https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access.\n",
"\n",
"```\n",
"\n",
"This exception will indicate that the deployment was blocked due to a policy violation."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D6ZikIbpYPk5"
},
"source": [
"#### Missing EULA (End-User License Agreement)\n",
"\n",
"You're trying to deploy the model `publishers/meta/models/llama3-2@llama-3.2-90b-vision`. The model name is correct, the model exists, you have sufficient quota, and no organizational policies are blocking it.\n",
"\n",
"However, you haven't accepted the model's EULA. Many models, especially those from third-party publishers (like Meta in this case), require you to agree to their terms of service before you can use them.\n",
"\n",
"The `model.deploy()` call will raise a Python exception. This exception will indicate that the deployment failed because the EULA hasn't been accepted.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "PP-ZZUH85LzV"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\n",
" \"publishers/meta/models/llama3-2@llama-3.2-90b-vision\"\n",
" )\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "MCoNmRJubrSi"
},
"source": [
"#### Request Hugging Face gated model without access token\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j7KbEY8aPoaE"
},
"source": [
"You are trying to deploy `black-forest-labs/flux.1-dev`, which, is a gated model on the Hugging Face Hub. This means you need a Hugging Face Hub account, acceptance of the model's license, and a valid read-only access token to download the model weights. In this scenario, you're missing the necessary access token.\n",
"\n",
"The `model.deploy()` call will raise a Python exception. The exception will signal that authentication with the Hugging Face Hub failed due to a missing token."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ktX2GNEy6hP4"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\"black-forest-labs/FLUX.1-dev\")\n",
" endpoint = model.deploy()\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VK67i0I2fBvb"
},
"source": [
"#### Request Hugging Face gated model with invalid access token\n",
"\n",
"You're attempting to deploy the gated model `black-forest-labs/FLUX.1-dev`. You're explicitly providing a Hugging Face access token via the hugging_face_access_token argument to the deploy method. However, the token `invalid-token` is, as the name suggests, not valid.\n",
"\n",
"The `model.deploy()` call will definitely raise an exception. The exception type will be similar to the previous missing token case, indicating an authentication failure with the Hugging Face Hub."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qTcDSSgG8NiR"
},
"outputs": [],
"source": [
"try:\n",
" model = model_garden.OpenModel(\"black-forest-labs/FLUX.1-dev\")\n",
" endpoint = model.deploy(hugging_face_access_token=\"invalid-token\")\n",
"except Exception as e:\n",
" print(f\"Error: {e}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2a4e033321ad"
},
"source": [
"## Cleaning up"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "N5UrKKMifeGi"
},
"outputs": [],
"source": [
"delete_endpoints = True\n",
"\n",
"if delete_endpoints:\n",
" gemma_endpoint.delete(force=True)\n",
" sd_endpoint.delete(force=True)"
]
}
],
"metadata": {
"colab": {
"name": "get_started_with_model_garden_sdk.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}