gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb (693 lines of code) (raw):
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ijGzTHJJUCPY"
},
"outputs": [],
"source": [
"# Copyright 2024 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "VEqbX8OhE8y9"
},
"source": [
"# Getting Started with RAGAS & Gemini API in Vertex AI\n",
"\n",
"<table align=\"left\">\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fretrieval-augmented-generation%2Frag-evaluation%2Fragas_with_gemini.ipynb\">\n",
" <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
"</table>\n",
"\n",
"<div style=\"clear: both;\"></div>\n",
"\n",
"<b>Share to:</b>\n",
"\n",
"<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
"</a>\n",
"\n",
"<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/retrieval-augmented-generation/rag-evaluation/ragas_with_gemini.ipynb\" target=\"_blank\">\n",
" <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
"</a> "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e57d479c5581"
},
"source": [
"| | |\n",
"|-|-|\n",
"| Author(s) | [Aditya Rane](https://github.com/Adi8885) |"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CkHPv2myT2cx"
},
"source": [
"## Overview\n",
"\n",
"### [RAGAS](https://docs.ragas.io/en/stable/)\n",
"\n",
"Ragas is a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines. RAG denotes a class of LLM applications that use external data to augment the LLM's context. There are existing tools and frameworks that help you build these pipelines but evaluating it and quantifying your pipeline performance can be hard. This is where Ragas (RAG Assessment) comes in.\n",
"\n",
"### Gemini\n",
"\n",
"Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases. The Gemini API gives you access to the Gemini models.\n",
"\n",
"### Gemini API in Vertex AI\n",
"\n",
"The Gemini API in Vertex AI provides a unified interface for interacting with Gemini models.\n",
"\n",
"You can interact with the Gemini API using the following methods:\n",
"\n",
"- Use the [Vertex AI Studio](https://cloud.google.com/generative-ai-studio) for quick testing and command generation\n",
"- Use cURL commands\n",
"- Use the Vertex AI SDK\n",
"\n",
"This notebook focuses on using the **Gemini model with RAGAS**\n",
"\n",
"For more information, see the [Generative AI on Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview) documentation."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DrkcqHrrwMAo"
},
"source": [
"### Objectives\n",
"\n",
"In this notebook we will focus on using the Gemini API in Vertex AI with RAGAS\n",
"We will use the Gemini (`gemini-2.0-flash`) model for Q&A evaluation.\n",
"\n",
"You will complete the following tasks:\n",
"\n",
"- Install the Vertex AI SDK for Python\n",
"- Use the Gemini API in Vertex AI to interact with each model\n",
" - Gemini (`gemini-2.0-flash`) model:\n",
" - Q&A Generation\n",
" - Evaluate Q&A performance with RAGAS"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "C9nEPojogw-g"
},
"source": [
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"- Vertex AI\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r11Gu7qNgx1p"
},
"source": [
"## Getting Started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "No17Cw5hgx12"
},
"source": [
"### Install Vertex AI SDK for Python"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tFy3H3aPgx12"
},
"outputs": [],
"source": [
"%pip install --user install ragas==0.1.6 \\\n",
"datasets==2.18.0 \\\n",
"langchain==0.1.14 \\\n",
"langchain-google-vertexai==1.0.5 \\\n",
"langchain-chroma==0.1.1 \\\n",
"chromadb==0.5.0 \\\n",
"pypdf==4.2.0 \\"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j7UyNVSiyQ96"
},
"source": [
"### Restart current runtime\n",
"\n",
"To use the newly installed packages in this Jupyter runtime, it is recommended to restart the runtime. Run the following cell to restart the current kernel.\n",
"\n",
"The restart process might take a minute or so."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "YmY9HVVGSBW5"
},
"outputs": [],
"source": [
"import IPython\n",
"\n",
"app = IPython.Application.instance()\n",
"app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4TCvupe_UQvS"
},
"source": [
"After the restart is complete, continue to the next step."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "EXQZrM5hQeKb"
},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
"<b>⚠️ Wait for the kernel to finish restarting before you continue. ⚠️</b>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jXHfaVS66_01"
},
"source": [
"## Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "lslYAvw37JGQ"
},
"outputs": [],
"source": [
"from datasets import Dataset\n",
"from langchain.chains import RetrievalQA\n",
"from langchain.prompts import PromptTemplate\n",
"from langchain_chroma import Chroma\n",
"from langchain_community.document_loaders import PyPDFLoader\n",
"from langchain_google_vertexai import VertexAI, VertexAIEmbeddings\n",
"import pandas as pd\n",
"from ragas import evaluate\n",
"\n",
"# Important to make Gemini Work with RAGAS\n",
"from ragas.llms.base import LangchainLLMWrapper\n",
"from ragas.metrics import (\n",
" answer_relevancy,\n",
" answer_similarity,\n",
" context_precision,\n",
" context_recall,\n",
")\n",
"from ragas.metrics.critique import harmfulness\n",
"import vertexai"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "812bdbe00afb"
},
"outputs": [],
"source": [
"# TODO(developer): Update the below lines\n",
"PROJECT_ID = \"<your_project>\"\n",
"LOCATION = \"<your_region>\"\n",
"\n",
"vertexai.init(project=PROJECT_ID, location=LOCATION)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4437b7608c8e"
},
"source": [
"## Use Vertex AI models\n",
"\n",
"The [Gemini](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/overview) models are designed to handle multimodal inputs."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "99c945d2fe66"
},
"outputs": [],
"source": [
"# Load the Gemini model\n",
"llm = VertexAI(model_name=\"gemini-2.0-flash\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d28ebabd3f38"
},
"source": [
"The [Vertex AI Embeddings](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings) models are designed to convert text to dense vector representations"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "356471d32ab3"
},
"outputs": [],
"source": [
"# Load Embeddings Models\n",
"embeddings = VertexAIEmbeddings(model_name=\"text-embedding-005\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f2aa34a39acb"
},
"source": [
"## Create a local Vector DB\n",
"### Load the document"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "398e911a765b"
},
"outputs": [],
"source": [
"# source document\n",
"document_uri = \"https://arxiv.org/pdf/1706.03762\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "16aa29d18990"
},
"outputs": [],
"source": [
"# use PyPDF loaded to read and chunk the input document\n",
"loader = PyPDFLoader(document_uri)\n",
"docs = loader.load_and_split()\n",
"\n",
"# Verify if pages are loaded correctly\n",
"docs[0]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cdddcf1fe8d1"
},
"source": [
"### Create local Vector DB"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6648f7c42278"
},
"outputs": [],
"source": [
"# Create an in-memory Vector DB using Chroma\n",
"vectordb = Chroma.from_documents(docs, embeddings)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "781b31cf93f9"
},
"outputs": [],
"source": [
"# Set Vector DB as retriever\n",
"retriever = vectordb.as_retriever()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "05c304041ced"
},
"source": [
"### Create Q&A Chain"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "635c5f3b7161"
},
"outputs": [],
"source": [
"# Create Q&A template for the Gemini Model\n",
"template = \"\"\"You task is to answer questions related documents.\n",
"Use the following context to answer the question at the end.\n",
"{context}\n",
"\n",
"Answers should be crisp.\n",
"\n",
"Question: {question}\n",
"Helpful Answer:\"\"\"\n",
"\n",
"# Create a prompt template for the q&a chain\n",
"PROMPT = PromptTemplate(\n",
" template=template,\n",
" input_variables=[\"context\", \"question\"],\n",
")\n",
"\n",
"# Pass prompts to q&a chain\n",
"chain_type_kwargs = {\"prompt\": PROMPT}\n",
"\n",
"# Retriever arguments\n",
"retriever.search_kwargs = {\"k\": 3}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7d7839c6d61a"
},
"outputs": [],
"source": [
"# Setup a RetrievalQA Chain\n",
"qa = RetrievalQA.from_chain_type(\n",
" llm=llm,\n",
" chain_type=\"stuff\",\n",
" retriever=retriever,\n",
" return_source_documents=True,\n",
" chain_type_kwargs=chain_type_kwargs,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "23879232b593"
},
"outputs": [],
"source": [
"# Test the chain with a sample question\n",
"query = \"Who are the authors of paper on Attention is all you need\"\n",
"result = qa({\"query\": query})\n",
"result"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "01ed0ed9fabe"
},
"source": [
"## Evaluation\n",
"### Create the evaluation set"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8d520de4ed86"
},
"outputs": [],
"source": [
"# Evaluation set with questions and ground_truth\n",
"questions = [\n",
" \"Who is the author of paper Attention is all you need\",\n",
" \"What architecture is proposed in paper titled Attention is all you need?\",\n",
"]\n",
"ground_truth = [\n",
" \"The authors of the paper 'Attention is all you need' are:\\n\\n* Ashish Vaswani\\n* Noam Shazeer\\n* Niki Parmar\\n* Jakob Uszkoreit\\n* Llion Jones\\n* Aidan N. Gomez\\n* Łukasz Kaiser\\n* Illia Polosukhin\",\n",
" \"Transformers architecture\",\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8e1c60b44b2c"
},
"source": [
"### Run the [Q&A chain](#create-qa-chain) on evaluation dataset "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "eae0927ed7e5"
},
"outputs": [],
"source": [
"contexts = []\n",
"answers = []\n",
"\n",
"# Generate contexts and answers for each question\n",
"for query in questions:\n",
" result = qa({\"query\": query})\n",
" contexts.append(\n",
" [document.page_content for document in result.get(\"source_documents\")]\n",
" )\n",
" answers.append(result.get(\"result\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "21a94628a922"
},
"outputs": [],
"source": [
"# Convert into a dataset and prepare for consumtion by RAGAS API\n",
"data = {\n",
" \"question\": questions,\n",
" \"contexts\": contexts,\n",
" \"ground_truth\": ground_truth,\n",
" \"answer\": answers,\n",
"}\n",
"\n",
"dataset = Dataset.from_dict(data)\n",
"dataset"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1b5623448cf9"
},
"outputs": [],
"source": [
"# Compile list of RAGAS Metrics\n",
"metrics = [\n",
" answer_relevancy,\n",
" context_recall,\n",
" context_precision,\n",
" harmfulness,\n",
" answer_similarity,\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5b3cc1516432"
},
"source": [
"## IMPORTANT : Gemini with RAGAS\n",
"> RAGAS is designed to work with OpenAI Models by default. We must set a few attributes to make it work with Gemini"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "b96ff0b8f2a7"
},
"outputs": [],
"source": [
"class RAGASVertexAIEmbeddings(VertexAIEmbeddings):\n",
" \"\"\"Wrapper for RAGAS\"\"\"\n",
"\n",
" async def embed_text(self, text: str) -> list[float]:\n",
" \"\"\"Embeds a text for semantics similarity\"\"\"\n",
" return self.embed([text], 1, \"SEMANTIC_SIMILARITY\")[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5c409c0d7f4f"
},
"outputs": [],
"source": [
"# Wrapper to make RAGAS work with Gemini and Vertex AI Embeddings Models\n",
"embeddings = RAGASVertexAIEmbeddings(model_name=\"text-embedding-005\")\n",
"ragas_llm = LangchainLLMWrapper(llm)\n",
"\n",
"for m in metrics:\n",
" # change LLM for metric\n",
" m.__setattr__(\"llm\", ragas_llm)\n",
"\n",
" # check if this metric needs embeddings\n",
" if hasattr(m, \"embeddings\"):\n",
" # if so change with Vertex AI Embeddings\n",
" m.__setattr__(\"embeddings\", embeddings)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "f8955e5d6a06"
},
"source": [
"### Run the RAGAS Evaluation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "e67d15b8c33a"
},
"outputs": [],
"source": [
"# Run the evaluation on every row of the dataset\n",
"result_set = []\n",
"for i in range(len(dataset)):\n",
" result = evaluate(\n",
" dataset=Dataset.from_dict(dataset[i : i + 1]),\n",
" metrics=metrics,\n",
" raise_exceptions=False,\n",
" )\n",
" result_set.append(result.to_pandas())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "5eed0c363bfc"
},
"outputs": [],
"source": [
"# View results in Pandas DataFrame\n",
"results_df = pd.concat(result_set)\n",
"results_df"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e79872abe17e"
},
"source": [
"# Conclusion\n",
"\n",
"In this notebook, you learned:\n",
"\n",
"1. RAGAS - Framework for evaluation .\n",
"2. Making RAGAS Work with Gemini API in Vertex AI"
]
}
],
"metadata": {
"colab": {
"name": "ragas_with_gemini.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}