gemini/function-calling/function_calling_data_structures.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ijGzTHJJUCPY"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VEqbX8OhE8y9"
      },
      "source": [
        "# Working with Data Structures and Schemas in Gemini Function Calling\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Ffunction-calling%2Ffunction_calling_data_structures.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/function-calling/function_calling_data_structures.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/function-calling/function_calling_data_structures.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZNJC1SkrsJY3"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Kristopher Overholt](https://github.com/koverholt) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CkHPv2myT2cx"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases. The Gemini API gives you access to the Gemini models.\n",
        "\n",
        "[Function Calling](https://cloud.google.com/vertex-ai/docs/generative-ai/multimodal/function-calling) in Gemini lets you create a description of a function in your code, then pass that description to a language model in a request. The response from the model includes the name of a function that matches the description and the arguments to call it with.\n",
        "\n",
        "In this tutorial, you'll learn how to work with various data structures within Gemini Function Calling, including:\n",
        "    \n",
        "- Single parameter\n",
        "- Multiple parameters\n",
        "- Lists of parameters\n",
        "- Nested parameters and data structures"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r11Gu7qNgx1p"
      },
      "source": [
        "## Getting Started\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Google Gen AI SDK"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
        "\n",
        "The restart might take a minute or longer. After it's restarted, continue to the next step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information and create client\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
        "\n",
        "from google import genai\n",
        "\n",
        "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jXHfaVS66_01"
      },
      "source": [
        "## Code Examples\n",
        "\n",
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lslYAvw37JGQ"
      },
      "outputs": [],
      "source": [
        "from google.genai.types import FunctionDeclaration, GenerateContentConfig, Tool"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bfy0nopcsJY5"
      },
      "source": [
        "### Initialize model\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hvQy2EObsJY5"
      },
      "outputs": [],
      "source": [
        "MODEL_ID = \"gemini-2.0-flash\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3KHAr6BsJY6"
      },
      "source": [
        "### Example: Single parameter\n",
        "\n",
        "Let's say that you want to extract a location from a prompt to help a user navigate to their desired destination.\n",
        "\n",
        "You can build out a simple schema for a function that takes a single parameter as an input:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ElbXcrGWsJY6"
      },
      "outputs": [],
      "source": [
        "get_destination = FunctionDeclaration(\n",
        "    name=\"get_destination\",\n",
        "    description=\"Get directions to a destination\",\n",
        "    parameters={\n",
        "        \"type\": \"object\",\n",
        "        \"properties\": {\n",
        "            \"destination\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Destination that the user wants to go to\",\n",
        "            },\n",
        "        },\n",
        "    },\n",
        ")\n",
        "\n",
        "destination_tool = Tool(\n",
        "    function_declarations=[get_destination],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_u_cPW7sJY6"
      },
      "source": [
        "Now you can send a prompt with a destination, and the model will return structured data with a single key/value pair:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aD4UJ6BcsJY6"
      },
      "outputs": [],
      "source": [
        "prompt = \"I'd like to travel to Paris\"\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=prompt,\n",
        "    config=GenerateContentConfig(temperature=0, tools=[destination_tool]),\n",
        ")\n",
        "\n",
        "response.function_calls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7jimk65sJY7"
      },
      "source": [
        "### Example: Multiple parameters\n",
        "\n",
        "What if you want the function call to return more than one parameter?\n",
        "\n",
        "You can build out a simple schema for a function that takes multiple parameters as an input:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OTaA258isJY7"
      },
      "outputs": [],
      "source": [
        "get_destination_params = FunctionDeclaration(\n",
        "    name=\"get_destination_params\",\n",
        "    description=\"Get directions to a destination\",\n",
        "    parameters={\n",
        "        \"type\": \"object\",\n",
        "        \"properties\": {\n",
        "            \"destination\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Destination that the user wants to go to\",\n",
        "            },\n",
        "            \"mode_of_transportation\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Mode of transportation to use\",\n",
        "            },\n",
        "            \"departure_time\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"Time that the user will leave for the destination\",\n",
        "            },\n",
        "        },\n",
        "    },\n",
        ")\n",
        "\n",
        "destination_tool = Tool(\n",
        "    function_declarations=[get_destination_params],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xS-qg3udsJY7"
      },
      "source": [
        "Now you can send a prompt with a destination, and the model will return structured data with a single key/value pair:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2gm8YuoesJY7"
      },
      "outputs": [],
      "source": [
        "prompt = \"I'd like to travel to Paris by train and leave at 9:00 am\"\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=prompt,\n",
        "    config=GenerateContentConfig(temperature=0, tools=[destination_tool]),\n",
        ")\n",
        "\n",
        "response.function_calls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y-ZP3TKrsJY7"
      },
      "source": [
        "### Example: Lists of parameters\n",
        "\n",
        "What if you want the function call to return an array or list of parameters within an object?\n",
        "\n",
        "For example, you might want to call an API to get the geocoded coordinates of several different locations within a single prompt.\n",
        "\n",
        "In that case, you can build out a schema for a function that takes an array as an input:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jGMeCeftsJY7"
      },
      "outputs": [],
      "source": [
        "get_multiple_location_coordinates = FunctionDeclaration(\n",
        "    name=\"get_location_coordinates\",\n",
        "    description=\"Get coordinates of multiple locations\",\n",
        "    parameters={\n",
        "        \"type\": \"object\",\n",
        "        \"properties\": {\n",
        "            \"locations\": {\n",
        "                \"type\": \"array\",\n",
        "                \"description\": \"A list of locations\",\n",
        "                \"items\": {\n",
        "                    \"description\": \"Components of the location\",\n",
        "                    \"type\": \"object\",\n",
        "                    \"properties\": {\n",
        "                        \"point_of_interest\": {\n",
        "                            \"type\": \"string\",\n",
        "                            \"description\": \"Name or type of point of interest\",\n",
        "                        },\n",
        "                        \"city\": {\"type\": \"string\", \"description\": \"City\"},\n",
        "                        \"country\": {\"type\": \"string\", \"description\": \"Country\"},\n",
        "                    },\n",
        "                    \"required\": [\n",
        "                        \"point_of_interest\",\n",
        "                        \"city\",\n",
        "                        \"country\",\n",
        "                    ],\n",
        "                },\n",
        "            }\n",
        "        },\n",
        "    },\n",
        ")\n",
        "\n",
        "geocoding_tool = Tool(\n",
        "    function_declarations=[get_multiple_location_coordinates],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p2X7KXuYsJY8"
      },
      "source": [
        "Now we'll send a prompt with a few different locations and points of interest:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8YtBm7-XsJY8"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"\n",
        "    I'd like to get the coordinates for\n",
        "    the Eiffel tower in Paris,\n",
        "    the statue of liberty in New York,\n",
        "    and Port Douglas near the Great Barrier Reef.\n",
        "\"\"\"\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=prompt,\n",
        "    config=GenerateContentConfig(temperature=0, tools=[geocoding_tool]),\n",
        ")\n",
        "\n",
        "response.function_calls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D8giW_8OsJY8"
      },
      "source": [
        "Note that the generative model populated values for all of the parameters for a given location since all three parameters are required."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jEHIc8x6sJY8"
      },
      "source": [
        "### Example: Nested parameters and data structures\n",
        "\n",
        "What if you want the function call to include nested parameters or other complex data structures?\n",
        "\n",
        "You might want to send a command to create a product listing based on a few sentences that include the product details.\n",
        "\n",
        "In that case, you can build out a schema for a function that takes nested data structures as an input:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "27PuX0-fsJY8"
      },
      "outputs": [],
      "source": [
        "create_product_listing = FunctionDeclaration(\n",
        "    name=\"create_product_listing\",\n",
        "    description=\"Create a product listing using the details provided by the user.\",\n",
        "    parameters={\n",
        "        \"type\": \"object\",\n",
        "        \"properties\": {\n",
        "            \"product\": {\n",
        "                \"type\": \"object\",\n",
        "                \"properties\": {\n",
        "                    \"name\": {\"type\": \"string\"},\n",
        "                    \"price\": {\"type\": \"number\"},\n",
        "                    \"category\": {\"type\": \"string\"},\n",
        "                    \"description\": {\"type\": \"string\"},\n",
        "                },\n",
        "            }\n",
        "        },\n",
        "    },\n",
        ")\n",
        "\n",
        "product_listing_tool = Tool(\n",
        "    function_declarations=[create_product_listing],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k8sKF69fsJY8"
      },
      "source": [
        "Now we'll send a prompt with a few different locations and attributes:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Pt-i-srosJY8"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"Create a listing for noise-canceling headphones for $149.99.\n",
        "These headphones create a distraction-free environment.\"\"\"\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=prompt,\n",
        "    config=GenerateContentConfig(temperature=0, tools=[product_listing_tool]),\n",
        ")\n",
        "\n",
        "response.function_calls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GpDvGrmtsJY8"
      },
      "source": [
        "And you're done! You successfully generated various types of data structures, including a single parameter, multiple parameters, a list of parameters, and nested parameters. Try another notebook to continue exploring other functionality in the Gemini API!"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "function_calling_data_structures.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}