gemini/getting-started/intro_gemini_curl.ipynb (907 lines of code) (raw):
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijGzTHJJUCPY"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ZPC2X_a9ErW7"
},
"source": [
"# Getting Started with the Gemini API in Vertex AI with cURL / REST API\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fgetting-started%2Fintro_gemini_curl.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
" </a>\n",
" </td> \n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/getting-started/intro_gemini_curl.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://goo.gle/4jeQxSk\">\n",
" <img width=\"32px\" src=\"https://cdn.qwiklabs.com/assets/gcp_cloud-e3a77215f0b8bfa9b3f611c0d2208c7e8708ed31.svg\" alt=\"Google Cloud logo\"><br> Open in Cloud Skills Boost\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/getting-started/intro_gemini_curl.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f0cc0f48513b"
},
"source": [
"| Author(s) |\n",
"| --- |\n",
"| [Eric Dong](https://github.com/gericdong) |\n",
"| [Polong Lin](https://github.com/polong-lin) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "axauUzNXEl_R"
},
"source": [
"## Overview\n",
"\n",
"**YouTube Video: Introduction to Gemini on Vertex AI**\n",
"\n",
"<a href=\"https://www.youtube.com/watch?v=YfiLUpNejpE&list=PLIivdWyY5sqJio2yeg1dlfILOUO2FoFRx\" target=\"_blank\">\n",
" <img src=\"https://img.youtube.com/vi/YfiLUpNejpE/maxresdefault.jpg\" alt=\"Introduction to Gemini on Vertex AI\" width=\"500\">\n",
"</a>\n",
"\n",
"In this tutorial, you learn how to use the Vertex AI REST API with cURL commands to interact with the Gemini 2.0 Flash model.\n",
"\n",
"You will complete the following tasks:\n",
"\n",
"- Text generation\n",
"- Streaming text generation\n",
"- Chat\n",
"- Function Calling\n",
"- Multimodal Input\n",
"- Controlled generation\n",
"- Search as a tool\n",
"- Code execution"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "wJf9sLIIEl_S"
},
"source": [
"### Costs\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"- Vertex AI\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D50ekWXjEl_S"
},
"source": [
"## Getting Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5f7c203ffaa1"
},
"source": [
"### Install required libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "4e66b2f6d36f"
},
"outputs": [],
"source": [
"%%capture\n",
"\n",
"!sudo apt install -q jq"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "dmWOrTJ3gx13"
},
"source": [
"### Authenticate your notebook environment (Colab only)\n",
"\n",
"If you are running this notebook on Google Colab, run the following cell to authenticate your environment.\n",
"\n",
"This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "NyKGtVQjgx13"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"# Additional authentication is required for Google Colab\n",
"if \"google.colab\" in sys.modules:\n",
" # Authenticate user to Google Cloud\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "O6ZGaZlxP9L0"
},
"source": [
"### Set Google Cloud project\n",
"\n",
"To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
"\n",
"Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "u8IivOG5SqY6"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
"\n",
"os.environ[\"GOOGLE_CLOUD_PROJECT\"] = PROJECT_ID\n",
"os.environ[\"GOOGLE_CLOUD_REGION\"] = LOCATION"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "854fbf388e2b"
},
"source": [
"## Use the Gemini 2.0 Flash model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7eeb063ac6d4"
},
"outputs": [],
"source": [
"MODEL_ID = \"gemini-2.0-flash\"\n",
"API_HOST = f\"{LOCATION}-aiplatform.googleapis.com\"\n",
"\n",
"os.environ[\"API_ENDPOINT\"] = (\n",
" f\"{API_HOST}/v1/projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/{MODEL_ID}\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0ZZUVBSzc0cR"
},
"source": [
"## Text generation\n",
"\n",
"The `generateContent` method can handle a wide variety of use cases, including multi-turn chat and multimodal input, depending on what the underlying model supports. In this example, you send a text prompt and request the model response in text."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1979afec8834"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d '{\n",
" \"contents\": {\n",
" \"role\": \"USER\",\n",
" \"parts\": { \"text\": \"Why is the sky blue?\" },\n",
" },\n",
" \"generation_config\": {\n",
" \"response_modalities\": \"TEXT\",\n",
" },\n",
" }' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].text\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "27701e417da6"
},
"source": [
"### Streaming\n",
"\n",
"The Gemini API provides a streaming response mechanism. With this approach, you don't need to wait for the complete response; you can start processing fragments as soon as they're accessible."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "rzkCij_iS0we"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:streamGenerateContent \\\n",
" \\\n",
" -d '{\n",
" \"contents\": {\n",
" \"role\": \"USER\",\n",
" \"parts\": { \"text\": \"Why is the sky blue?\" }\n",
" }\n",
" }' 2>/dev/null >response.json\n",
"\n",
"jq -r \".[] | .candidates[] | .content.parts[].text\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3e56BV7PH9t8"
},
"source": [
"### Model parameters\n",
"\n",
"Every prompt you send to the model includes parameter values that control how the model generates a response. The model can generate different results for different parameter values. You can experiment with different model parameters to see how the results change. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Px8hSHhiH9t8"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d '{\n",
" \"contents\": {\n",
" \"role\": \"USER\",\n",
" \"parts\": [\n",
" {\"text\": \"Tell me a story.\"}\n",
" ]\n",
" },\n",
" \"generation_config\": {\n",
" \"temperature\": 0.2,\n",
" \"top_p\": 0.1,\n",
" \"top_k\": 16,\n",
" \"max_output_tokens\": 2048,\n",
" \"candidate_count\": 1,\n",
" \"stop_sequences\": []\n",
" },\n",
" \"safety_settings\": {\n",
" \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
" \"threshold\": \"BLOCK_LOW_AND_ABOVE\"\n",
" }\n",
" }' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].text\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "I4-XhmPn_Pb-"
},
"source": [
"### Chat\n",
"\n",
"The Gemini API supports natural multi-turn conversations and is ideal for text tasks that require back-and-forth interactions.\n",
"\n",
"Specify the `role` field only if the content represents a turn in a conversation. You can set `role` to one of the following values: `user`, `model`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "YqSQSK-K-KVU"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d '{\n",
" \"contents\": [\n",
" {\n",
" \"role\": \"user\",\n",
" \"parts\": [\n",
" { \"text\": \"Hello\" }\n",
" ]\n",
" },\n",
" {\n",
" \"role\": \"model\",\n",
" \"parts\": [\n",
" { \"text\": \"Hello! I am glad you could both make it.\" }\n",
" ]\n",
" },\n",
" {\n",
" \"role\": \"user\",\n",
" \"parts\": [\n",
" { \"text\": \"So what is the first order of business?\" }\n",
" ]\n",
" }\n",
" ]\n",
" }' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].text\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5f0f5fe3b331"
},
"source": [
"### Function calling\n",
"\n",
"Function calling lets you create a description of a function in their code, then pass that description to a language model in a request. This sample is an example of passing in a description of a function that returns information about where a movie is playing. Several function declarations are included in the request, such as `find_movies` and `find_theaters`.\n",
"\n",
"Learn more about [function calling](https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/function-calling)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "680b11b0ba4c"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d '{\n",
" \"contents\": {\n",
" \"role\": \"user\",\n",
" \"parts\": {\n",
" \"text\": \"Which theaters in Mountain View show Barbie movie?\"\n",
" }\n",
" },\n",
" \"tools\": [\n",
" {\n",
" \"function_declarations\": [\n",
" {\n",
" \"name\": \"find_movies\",\n",
" \"description\": \"find movie titles currently playing in theaters based on any description, genre, title words, etc.\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"location\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The city and state, e.g. San Francisco, CA or a zip code e.g. 95616\"\n",
" },\n",
" \"description\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Any kind of description including category or genre, title words, attributes, etc.\"\n",
" }\n",
" },\n",
" \"required\": [\n",
" \"description\"\n",
" ]\n",
" }\n",
" },\n",
" {\n",
" \"name\": \"find_theaters\",\n",
" \"description\": \"find theaters based on location and optionally movie title which are is currently playing in theaters\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"location\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The city and state, e.g. San Francisco, CA or a zip code e.g. 95616\"\n",
" },\n",
" \"movie\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Any movie title\"\n",
" }\n",
" },\n",
" \"required\": [\n",
" \"location\"\n",
" ]\n",
" }\n",
" },\n",
" {\n",
" \"name\": \"get_showtimes\",\n",
" \"description\": \"Find the start times for movies playing in a specific theater\",\n",
" \"parameters\": {\n",
" \"type\": \"object\",\n",
" \"properties\": {\n",
" \"location\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"The city and state, e.g. San Francisco, CA or a zip code e.g. 95616\"\n",
" },\n",
" \"movie\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Any movie title\"\n",
" },\n",
" \"theater\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Name of theater\"\n",
" },\n",
" \"date\": {\n",
" \"type\": \"string\",\n",
" \"description\": \"Date for requested showtime\"\n",
" }\n",
" },\n",
" \"required\": [\n",
" \"location\",\n",
" \"movie\",\n",
" \"theater\",\n",
" \"date\"\n",
" ]\n",
" }\n",
" }\n",
" ]\n",
" }\n",
" ]\n",
"}' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].functionCall\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "R3g5n23lDtsN"
},
"source": [
"## Multimodal input\n",
"\n",
"Gemini is a multimodal model that supports adding image and video in text or chat prompts for a text response.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uTfL2DDch4Lp"
},
"source": [
"### Download an image from Google Cloud Storage"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KmtWSNLtJ7oD"
},
"outputs": [],
"source": [
"! gsutil cp \"gs://cloud-samples-data/generative-ai/image/320px-Felis_catus-cat_on_snow.jpg\" ./image.jpg"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BlyyaPgmhpyv"
},
"source": [
"### Generate text from a local image\n",
"\n",
"Specify the [base64](https://en.wikipedia.org/wiki/Base64) encoding of the image or video to include inline in the prompt and the `mime_type` field. The supported [MIME types](https://en.wikipedia.org/wiki/Media_type) for images include `image/png` and `image/jpeg`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-uqZ-RWdtdit"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"# Encode image data in base64\n",
"image_file=\"image.jpg\"\n",
"if [[ -f \"$image_file\" ]]; then\n",
" if command -v base64 &> /dev/null; then\n",
" # base64 is available\n",
" if [[ \"$(uname -s)\" == \"Darwin\" ]]; then\n",
" # macOS -b 0 to avoid line wrapping\n",
" data=$(base64 -b 0 -i \"$image_file\")\n",
" else\n",
" # Linux -w 0 to avoid line wrapping\n",
" data=$(base64 -w 0 \"$image_file\")\n",
" fi\n",
" else\n",
" echo \"Error: base64 command not found.\"\n",
" exit 1\n",
" fi\n",
"else\n",
" echo \"Error: Image file '$image_file' not found.\"\n",
" exit 1\n",
"fi\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d \"{\n",
" 'contents': {\n",
" 'role': 'USER',\n",
" 'parts': [\n",
" {\n",
" 'text': 'Is it a cat?'\n",
" },\n",
" {\n",
" 'inline_data': {\n",
" 'data': '${data}',\n",
" 'mime_type':'image/jpeg'\n",
" }\n",
" }\n",
" ]\n",
" }\n",
" }\" 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].text\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fKr-BklmhjgP"
},
"source": [
"### Generate text from an image on Google Cloud Storage\n",
"\n",
"Specify the Cloud Storage URI of the image to include in the prompt. The bucket that stores the file must be in the same Google Cloud project that's sending the request. You must also specify the `mime_type` field. The supported image MIME types include `image/png` and `image/jpeg`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "43pQE3_z3OjG"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d '{\n",
" \"contents\": {\n",
" \"role\": \"USER\",\n",
" \"parts\": [\n",
" {\n",
" \"text\": \"Describe this image\"\n",
" },\n",
" {\n",
" \"file_data\": {\n",
" \"mime_type\": \"image/png\",\n",
" \"file_uri\": \"gs://cloud-samples-data/generative-ai/image/320px-Felis_catus-cat_on_snow.jpg\"\n",
" }\n",
" }\n",
" ]\n",
" },\n",
" \"generation_config\": {\n",
" \"temperature\": 0.2,\n",
" \"top_p\": 0.1,\n",
" \"top_k\": 16,\n",
" \"max_output_tokens\": 2048,\n",
" \"candidate_count\": 1,\n",
" \"stop_sequences\": []\n",
" },\n",
" \"safety_settings\": {\n",
" \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
" \"threshold\": \"BLOCK_LOW_AND_ABOVE\"\n",
" }\n",
" }' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].text\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TVF4vHuBOD8N"
},
"source": [
"### Generate text from a video file\n",
"\n",
"Specify the Cloud Storage URI of the video to include in the prompt. The bucket that stores the file must be in the same Google Cloud project that's sending the request. You must also specify the `mime_type` field. The supported MIME types for video include `video/mp4`.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "F8kS5p0l_uHE"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d \\\n",
"'{\n",
" \"contents\": {\n",
" \"role\": \"USER\",\n",
" \"parts\": [\n",
" {\n",
" \"text\": \"Answer the following questions using the video only. What is the profession of the main person? What are the main features of the phone highlighted? Which city was this recorded in?\"\n",
" },\n",
" {\n",
" \"file_data\": {\n",
" \"mime_type\": \"video/mp4\",\n",
" \"file_uri\": \"gs://github-repo/img/gemini/multimodality_usecases_overview/pixel8.mp4\"\n",
" }\n",
" }\n",
" ]\n",
" }\n",
" }' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].text\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "133ddb1bc7ff"
},
"source": [
"### Controlled Generation\n",
"\n",
"Controlled generation allows you to define a response schema to specify the structure of a model's output, the field names, and the expected data type for each field."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "40db1e8d9061"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d '{\n",
" \"contents\": {\n",
" \"role\": \"user\",\n",
" \"parts\": {\n",
" \"text\": \"List a few popular cookie recipes.\"\n",
" }\n",
" },\n",
" \"generationConfig\": {\n",
" \"response_mime_type\": \"application/json\",\n",
" \"response_schema\": {\"type\": \"object\", \"properties\": {\"recipe_name\": {\"type\": \"string\"}}}\n",
" },\n",
"}' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].text\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ebcbb1533401"
},
"source": [
"## Search as a tool\n",
"\n",
"Using Grounding with Google Search, you can improve the accuracy and recency of responses from the model. Starting with Gemini 2.0, Google Search is available as a tool. This means that the model can decide when to use Google Search."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "a3d8a66bfb3c"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d '{\n",
" \"contents\": [\n",
" {\n",
" \"role\": \"user\",\n",
" \"parts\": [\n",
" {\n",
" \"text\": \"What is the weather today in San Jose CA?\"\n",
" },\n",
" ]\n",
" }\n",
" ],\n",
" \"tools\": {\n",
" \"google_search\": {}\n",
" },\n",
" \"generationConfig\": {\n",
" \"response_modalities\": \"TEXT\"\n",
" }\n",
"}' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[].text\" response.json\n",
"jq -r \".candidates[].groundingMetadata.groundingChunks\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "37223c8e3133"
},
"source": [
"### Code Execution\n",
"\n",
"The Gemini API code execution feature enables the model to generate and run Python code and learn iteratively from the results until it arrives at a final output."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7cebe2cd31c1"
},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"curl -X POST \\\n",
" -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n",
" -H \"Content-Type: application/json\" \\\n",
" https://${API_ENDPOINT}:generateContent \\\n",
" -d '{\n",
" \"contents\": {\n",
" \"role\": \"user\",\n",
" \"parts\": {\n",
" \"text\": \"Calculate 20th fibonacci number. Then find the nearest palindrome to it.\"\n",
" }\n",
" },\n",
" \"tools\": [\n",
" {\"code_execution\": {},}\n",
" ]\n",
"}' 2>/dev/null >response.json\n",
"\n",
"jq -r \".candidates[].content.parts[]\" response.json"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6301d8abe89d"
},
"source": [
"## What's next\n",
"\n",
"- Explore other notebooks in the [Google Cloud Generative AI GitHub repository](https://github.com/GoogleCloudPlatform/generative-ai).\n",
"- Explore AI models in [Model Garden](https://cloud.google.com/vertex-ai/generative-ai/docs/model-garden/explore-models)."
]
}
],
"metadata": {
"colab": {
"name": "intro_gemini_curl.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}