quickstarts/rest/Safety_REST.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2025 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeadDkMiISin"
      },
      "source": [
        "# Gemini API: Safety Quickstart"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEXQ3OwKIa-O"
      },
      "source": [
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/rest/Safety_REST.ipynb\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" height=30/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uOxMUKTxR-_j"
      },
      "source": [
        "The Gemini API has adjustable safety settings. This notebook walks you through how to use them. You'll write a prompt that's blocked, see the reason why, and then adjust the filters to unblock it.\n",
        "\n",
        "Safety is an important topic, and you can learn more with the links at the end of this notebook. For now just focus on the code."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gHYFrFPjSGNq"
      },
      "source": [
        "## Set up your API key\n",
        "\n",
        "If you want to quickly try out the Gemini API, you can use `curl` commands to call the methods in the REST API.\n",
        "\n",
        "This notebook contains `curl` commands you can run in Google Colab, or copy to your terminal.\n",
        "\n",
        "To run this notebook, your API key must be stored it in a Colab Secret named GOOGLE_API_KEY. If you are running in a different environment, you can store your key in an environment variable. See [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) to learn more."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "405ee147f509"
      },
      "outputs": [],
      "source": [
        "!apt install jq"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ab9ASynfcIZn"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from google.colab import userdata"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7b547b1d5cad"
      },
      "outputs": [],
      "source": [
        "os.environ['GOOGLE_API_KEY'] = userdata.get('GOOGLE_API_KEY')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 108,
      "metadata": {
        "id": "3defec89594e"
      },
      "outputs": [],
      "source": [
        "os.environ['UNSAFE_PROMPT'] = "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LZfoK3I3hu6V"
      },
      "source": [
        "## Prompt Feedback\n",
        "\n",
        "The result returned by the [Model.generate_content](https://ai.google.dev/api/python/google/generativeai/GenerativeModel#generate_content) method is a [genai.GenerateContentResponse](https://ai.google.dev/api/python/google/generativeai/types/GenerateContentResponse)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {
        "id": "6d9e5d84541c"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "echo '{\n",
        "      \"contents\": [{\n",
        "        \"parts\":[{\n",
        "          \"text\": \"'$UNSAFE_PROMPT'\"}]}]}' > request.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {
        "id": "2bcfnGEviwTI"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"promptFeedback\": {\n",
            "    \"blockReason\": \"SAFETY\",\n",
            "    \"safetyRatings\": [\n",
            "      {\n",
            "        \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
            "        \"probability\": \"NEGLIGIBLE\"\n",
            "      },\n",
            "      {\n",
            "        \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n",
            "        \"probability\": \"NEGLIGIBLE\"\n",
            "      },\n",
            "      {\n",
            "        \"category\": \"HARM_CATEGORY_HARASSMENT\",\n",
            "        \"probability\": \"MEDIUM\"\n",
            "      },\n",
            "      {\n",
            "        \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n",
            "        \"probability\": \"NEGLIGIBLE\"\n",
            "      }\n",
            "    ]\n",
            "  }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "\n",
        "curl \"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GOOGLE_API_KEY\" \\\n",
        "    -H 'Content-Type: application/json' \\\n",
        "    -X POST \\\n",
        "    -d @request.json  2> /dev/null | tee response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WR_2A_sxk8sK"
      },
      "source": [
        "Above you can see that the response object gives you safety feedback about the prompt in two ways:\n",
        "\n",
        "* The `prompt_feedback.safety_ratings` attribute contains a list of safety ratings for the input prompt.\n",
        "* If your prompt is blocked, `prompt_feedback.block_reason` field will explain why."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "72b4a8808bb9"
      },
      "source": [
        "If the prompt is blocked because of the safety ratings, you will not get any candidates in the response."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4672af98ac57"
      },
      "source": [
        "### Safety settings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2a6229f6d3a1"
      },
      "source": [
        "Adjust the safety settings and the prompt is no longer blocked:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 134,
      "metadata": {
        "id": "9c38561789c2"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "echo '{\n",
        "    \"safetySettings\": [\n",
        "        {'category': 7, 'threshold': 4}\n",
        "    ],\n",
        "    \"contents\": [{\n",
        "        \"parts\":[{\n",
        "          \"text\": \"'$UNSAFE_PROMPT'\"}]}]}' > request.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 143,
      "metadata": {
        "id": "338fb9a6af78"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "curl \"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=$GOOGLE_API_KEY\" \\\n",
        "    -H 'Content-Type: application/json' \\\n",
        "    -X POST \\\n",
        "    -d @request.json  2> /dev/null > response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86c560e0a641"
      },
      "source": [
        "With the new settings, the `blocked_reason` is no longer set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 147,
      "metadata": {
        "id": "0c2847c49262"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"safetyRatings\": [\n",
            "    {\n",
            "      \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
            "      \"probability\": \"NEGLIGIBLE\"\n",
            "    },\n",
            "    {\n",
            "      \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n",
            "      \"probability\": \"NEGLIGIBLE\"\n",
            "    },\n",
            "    {\n",
            "      \"category\": \"HARM_CATEGORY_HARASSMENT\",\n",
            "      \"probability\": \"MEDIUM\"\n",
            "    },\n",
            "    {\n",
            "      \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n",
            "      \"probability\": \"NEGLIGIBLE\"\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "%%bash \n",
        "\n",
        "jq .promptFeedback < response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "47298a4eef40"
      },
      "source": [
        "And a candidate response is returned."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "028febe8df68"
      },
      "outputs": [],
      "source": [
        "%%bash \n",
        "\n",
        "jq .candidates[0].content.parts[].text < response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ujVlQoC43N3B"
      },
      "source": [
        "You can check `response.text` for the response."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3d401c247957"
      },
      "source": [
        "### Candidate ratings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3d306960dffb"
      },
      "source": [
        "For a prompt that is not blocked, the response object contains a list of `candidate` objects (just 1 for now). Each candidate includes a `finish_reason`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 157,
      "metadata": {
        "id": "e49b53f69a2c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\"STOP\"\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "jq .candidates[0].finishReason < response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "badddf10089b"
      },
      "source": [
        "`FinishReason.STOP` means that the model finished its output normally.\n",
        "\n",
        "`FinishReason.SAFETY` means the candidate's `safety_ratings` exceeded the request's `safety_settings` threshold."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 158,
      "metadata": {
        "id": "2b60d9f96af0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[\n",
            "  {\n",
            "    \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
            "    \"probability\": \"NEGLIGIBLE\"\n",
            "  },\n",
            "  {\n",
            "    \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n",
            "    \"probability\": \"NEGLIGIBLE\"\n",
            "  },\n",
            "  {\n",
            "    \"category\": \"HARM_CATEGORY_HARASSMENT\",\n",
            "    \"probability\": \"NEGLIGIBLE\"\n",
            "  },\n",
            "  {\n",
            "    \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n",
            "    \"probability\": \"NEGLIGIBLE\"\n",
            "  }\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "jq .candidates[0].safetyRatings < response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n1UdbxVt3ysY"
      },
      "source": [
        "## Learning more\n",
        "\n",
        "Learn more with these articles on [safety guidance](https://ai.google.dev/docs/safety_guidance) and [safety settings](https://ai.google.dev/docs/safety_setting_gemini).\n",
        "\n",
        "## Useful API references\n",
        "\n",
        "- Safety settings can be set in the [genai.GenerativeModel](https://ai.google.dev/api/python/google/generativeai/GenerativeModel) constructor. They can also be passed on each request to [GenerativeModel.generate_content](https://ai.google.dev/api/python/google/generativeai/GenerativeModel#generate_content) or [ChatSession.send_message](https://ai.google.dev/api/python/google/generativeai/ChatSession?hl=en#send_message).\n",
        "- The [genai.GenerateContentResponse](https://ai.google.dev/api/python/google/generativeai/protos/GenerateContentResponse) returns [SafetyRatings](https://ai.google.dev/api/python/google/generativeai/protos/SafetyRating) for the prompt in the [GenerateContentResponse.prompt_feedback](https://ai.google.dev/api/python/google/generativeai/protos/GenerateContentResponse/PromptFeedback), and for each [Candidate](https://ai.google.dev/api/python/google/generativeai/protos/Candidate) in the `safety_ratings` attribute.\n",
        "- A [genai.protos.SafetySetting](https://ai.google.dev/api/python/google/generativeai/protos/SafetySetting)  contains: [genai.protos.HarmCategory](https://ai.google.dev/api/python/google/generativeai/protos/HarmCategory) and a [genai.protos.HarmBlockThreshold](https://ai.google.dev/api/python/google/generativeai/types/HarmBlockThreshold)\n",
        "- A [genai.protos.SafetyRating](https://ai.google.dev/api/python/google/generativeai/protos/SafetyRating) contains a [HarmCategory](https://ai.google.dev/api/python/google/generativeai/protos/HarmCategory) and a [HarmProbability](https://ai.google.dev/api/python/google/generativeai/types/HarmProbability)\n",
        "- The [genai.protos.HarmCategory](https://ai.google.dev/api/python/google/generativeai/protos/HarmCategory) enum includes both the categories for PaLM and Gemini models. The values allowed for Gemini models are `[7,8,9,10]`: `[HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_DANGEROUS_CONTENT]`.\n",
        "- When specifying enum values the SDK will accept the enum values themselves, or their integer or string representations. The SKD will also accept abbreviated string representations: `[\"HARM_CATEGORY_DANGEROUS_CONTENT\", \"DANGEROUS_CONTENT\", \"DANGEROUS\"]` are all valid. Strings are case insensitive.\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Safety_REST.ipynb",
      "toc_visible": true
    },
    "google": {
      "image_path": "/static/site-assets/images/docs/logo-python.svg",
      "keywords": [
        "examples",
        "gemini",
        "beginner",
        "googleai",
        "quickstart",
        "python",
        "text",
        "chat",
        "vision",
        "embed"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}