components/llm_service/notebooks/MG

{ "cells": [ { "cell_type": "markdown", "source": [ "# Vertex AI Model Garden - LLaMA2 (PEFT)\n", "\n", "## Overview\n", "\n", "This notebook demonstrates using of the existing prediction endpoint deployed to vertex AI Model Garden.\n", "\n", "You could check the ID of the existing prediction endpoint by navigating to [Vertex AI-> Online prediction](https://pantheon.corp.google.com/vertex-ai/online-prediction).\n", "\n", "\n", "\n" ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "## (Optional) Deploy Llama2 Model into Vertex AI\n", "\n", "Steps to deploy pre-built [LLaMA2 models](https://huggingface.co/meta-llama) with vLLM.\n", "\n", "### Setup Google Cloud project\n", "\n", "1. [Enable the Vertex AI API, Compute Engine API and Cloud Natural Language API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component,language.googleapis.com).\n", "\n", "1. [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs.\n", "\n", "1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create#iam-service-accounts-create-console) with `Vertex AI User` and `Storage Object Admin` roles for deploying fine tuned model to Vertex AI endpoint." ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "### Deploy Llama2\n", "\n", "Accept the model agreement to access the models:\n", "1. Navigate to the Vertex AI > Model Garden page in the Google Cloud console\n", "2. Find the LLaMA2 model card and click on \"VIEW DETAILS\"\n", "3. Review the agreement on the model card page.\n", "4. After clicking the agreement of LLaMA2, a Cloud Storage bucket containing LLaMA2 pretrained and finetuned models will be shared\n", "5. Deploy `Llama2-7B-chat-001` or `Llama2-13B-chat-001` model using One-click deploy\n", "6. Paste the Cloud Storage bucket link below which you have previously created and assign it to `VERTEX_AI_MODEL_GARDEN_LLAMA2`\n", "7. Note down the endpoint name, otherwise navigate to [Vertex AI-> Online prediction](https://pantheon.corp.google.com/vertex-ai/online-prediction) and copy Endpoint id from there." ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "## Using endpoint prediction" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 1, "outputs": [], "source": [ "import os\n", "import sys\n", "import vertexai\n", "from google.cloud import aiplatform\n", "sys.path.append(\"../../common/src\")\n", "sys.path.append(\"../src\")\n", "os.chdir(\"../src\")" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 2, "id": "24c68b1d-aea4-456c-ac69-2c7854f67bf1", "metadata": {}, "outputs": [], "source": [ "!export PROJECT_ID=\"gcp-mira-demo\"\n", "PROJECT_ID = \"gcp-mira-demo\"\n", "os.environ[\"PROJECT_ID\"] = PROJECT_ID\n", "os.environ[\"MODEL_GARDEN_LLAMA2_CHAT_ENDPOINT_ID\"] = \"3894402015861669888\"\n", "REGION = \"us-central1\" # @param {type:\"string\"}\n" ] }, { "cell_type": "code", "execution_count": 3, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO: [src/config.py:49 - <module>()] Namespace File not found, setting job namespace as default\n", "INFO: [src/config.py:84 - get_environ_flag()] ENABLE_GOOGLE_LLM = True\n", "INFO: [src/config.py:84 - get_environ_flag()] ENABLE_GOOGLE_MODEL_GARDEN = True\n", "INFO: [src/config.py:84 - get_environ_flag()] ENABLE_OPENAI_LLM = True\n", "INFO: [src/config.py:84 - get_environ_flag()] ENABLE_COHERE_LLM = True\n", "INFO: [src/config.py:191 - <module>()] LLM types loaded ['OpenAI-GPT3.5', 'OpenAI-GPT4', 'Cohere', 'VertexAI-Text', 'VertexAI-Chat', 'VertexAI-Chat', 'VertexAI-ModelGarden-LLAMA2-Chat']\n" ] } ], "source": [ "from services.llm_generate import model_garden_predict\n", "from services.llm_generate import llm_generate\n", "\n", "prompt = \"What is a Medicaid?\"" ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "Call prediction endpoint" ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 4, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO: [services/llm_generate.py:46 - llm_generate()] Generating text with an LLM given a prompt=What is a Medicaid?, llm_type=VertexAI-ModelGarden-LLAMA2-Chat\n", "INFO: [services/llm_generate.py:117 - model_garden_predict()] Generating text using Model Garden endpoint=[projects/gcp-mira-demo/locations/us-central1/endpoints/3894402015861669888], prompt=[What is a Medicaid?], parameters=[None.\n", "INFO: [services/llm_generate.py:140 - model_garden_predict()] Received response in 28 seconds from projects/63101149566/locations/us-central1/models/llama2-13b-chat-001-mg-one-click-deploy version=[1] with 1 prediction(s) = [Prompt:\n", "'What is a Medicaid?'\n", "Output:\n", " and 'How does it work?'\n", "\n", "Medicaid is a government program that provides health coverage to low-income individuals and families. It is jointly funded by the federal government and each state, and is administered by each state.\n", "\n", "To be eligible for Medicaid, you must meet certain income and asset requirements, which vary by state. Generally, you must have a very low income and limited assets to qualify.\n", "\n", "If you are eligible for Medicaid, you will be able to receive a wide range of health care services, including doctor visits, hospital stays, prescription drugs, and long-term care. Some states also cover certain services that are not covered by Medicare, such as dental and vision care.\n", "\n", "Medicaid is different from Medicare, which is a government program that provides health coverage to people who are 65 or older, or who have certain disabilities. Medicaid is also different from the Affordable Care Act (ACA), which is a government program that provides health coverage to people who do not have access to health insurance through their employer or another source.\n", "\n", "To apply for Medicaid, you will need to contact your state Medicaid office and complete an application. You will need to provide information about your income, assets, and other personal information. You may also need to provide proof of your identity and citizenship.\n", "\n", "Once you are approved for Medicaid, you will be able to receive health care services from any provider who accepts Medicaid. You will not need to pay any premiums or copays, as long as you meet the program's eligibility requirements.\n", "\n", "Overall, Medicaid is an important program that helps low-income individuals and families access the health care they need. If you are eligible, it can provide you with comprehensive health coverage and access to a wide range of health care services.] \n", "Prompt:\n", "'What is a Medicaid?'\n", "Output:\n", " and 'How does it work?'\n", "\n", "Medicaid is a government program that provides health coverage to low-income individuals and families. It is jointly funded by the federal government and each state, and is administered by each state.\n", "\n", "To be eligible for Medicaid, you must meet certain income and asset requirements, which vary by state. Generally, you must have a very low income and limited assets to qualify.\n", "\n", "If you are eligible for Medicaid, you will be able to receive a wide range of health care services, including doctor visits, hospital stays, prescription drugs, and long-term care. Some states also cover certain services that are not covered by Medicare, such as dental and vision care.\n", "\n", "Medicaid is different from Medicare, which is a government program that provides health coverage to people who are 65 or older, or who have certain disabilities. Medicaid is also different from the Affordable Care Act (ACA), which is a government program that provides health coverage to people who do not have access to health insurance through their employer or another source.\n", "\n", "To apply for Medicaid, you will need to contact your state Medicaid office and complete an application. You will need to provide information about your income, assets, and other personal information. You may also need to provide proof of your identity and citizenship.\n", "\n", "Once you are approved for Medicaid, you will be able to receive health care services from any provider who accepts Medicaid. You will not need to pay any premiums or copays, as long as you meet the program's eligibility requirements.\n", "\n", "Overall, Medicaid is an important program that helps low-income individuals and families access the health care they need. If you are eligible, it can provide you with comprehensive health coverage and access to a wide range of health care services.\n" ] } ], "source": [ "\n", "response = await llm_generate(llm_type=\"VertexAI-ModelGarden-LLAMA2-Chat\", prompt=prompt)\n", "print(response)" ], "metadata": { "collapsed": false } }, { "cell_type": "markdown", "source": [ "A custom way to modify prediction parameters, since performance depend on big extent on the output token size." ], "metadata": { "collapsed": false } }, { "cell_type": "code", "execution_count": 5, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "INFO: [services/llm_generate.py:117 - model_garden_predict()] Generating text using Model Garden endpoint=[projects/gcp-mira-demo/locations/us-central1/endpoints/3894402015861669888], prompt=[What is a Medicaid?], parameters=[{'max_tokens': 300, 'temperature': 0.2, 'top_p': 1.0, 'top_k': 10}.\n", "INFO: [services/llm_generate.py:140 - model_garden_predict()] Received response in 20 seconds from projects/63101149566/locations/us-central1/models/llama2-13b-chat-001-mg-one-click-deploy version=[1] with 1 prediction(s) = [Prompt:\n", "'What is a Medicaid?'\n", "Output:\n", " and 'How does it work?'\n", "\n", "Medicaid is a government program that provides health coverage to eligible individuals with low income and limited resources. It was created in 1965 as part of the Social Security Act and is administered by the Centers for Medicare and Medicaid Services (CMS), a division of the U.S. Department of Health and Human Services (HHS).\n", "\n", "Medicaid is a joint federal-state program, which means that both the federal government and each individual state contribute funds to the program. Each state has its own Medicaid program and sets its own eligibility requirements and benefits, but there are certain federal guidelines that all states must follow.\n", "\n", "To be eligible for Medicaid, an individual must meet certain income and resource requirements, which vary by state. Generally, individuals who are eligible for Medicaid are those with incomes below 138% of the federal poverty level (FPL) and have limited resources. Pregnant women, children, and individuals with disabilities may be eligible for Medicaid with higher income levels.\n", "\n", "Medicaid provides coverage for a wide range of health services, including:\n", "\n", "* Doctor visits and check-ups\n", "* Hospital care\n", "* Prescription drugs\n", "* Maternal and child health care\n", "* Mental health and substance abuse treatment] \n", "Prompt:\n", "'What is a Medicaid?'\n", "Output:\n", " and 'How does it work?'\n", "\n", "Medicaid is a government program that provides health coverage to eligible individuals with low income and limited resources. It was created in 1965 as part of the Social Security Act and is administered by the Centers for Medicare and Medicaid Services (CMS), a division of the U.S. Department of Health and Human Services (HHS).\n", "\n", "Medicaid is a joint federal-state program, which means that both the federal government and each individual state contribute funds to the program. Each state has its own Medicaid program and sets its own eligibility requirements and benefits, but there are certain federal guidelines that all states must follow.\n", "\n", "To be eligible for Medicaid, an individual must meet certain income and resource requirements, which vary by state. Generally, individuals who are eligible for Medicaid are those with incomes below 138% of the federal poverty level (FPL) and have limited resources. Pregnant women, children, and individuals with disabilities may be eligible for Medicaid with higher income levels.\n", "\n", "Medicaid provides coverage for a wide range of health services, including:\n", "\n", "* Doctor visits and check-ups\n", "* Hospital care\n", "* Prescription drugs\n", "* Maternal and child health care\n", "* Mental health and substance abuse treatment\n" ] } ], "source": [ "parameters = {\n", " \"max_tokens\": 300,\n", " \"temperature\": 0.2,\n", " \"top_p\": 1.0,\n", " \"top_k\": 10,\n", "}\n", "\n", "response = await model_garden_predict(aip_endpoint_name=os.environ[\"MODEL_GARDEN_LLAMA2_CHAT_ENDPOINT_ID\"], prompt=prompt, parameters=parameters)\n", "print(response)" ], "metadata": { "collapsed": false } } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 5 }

components/llm_service/notebooks/MG_Llama2Chat.ipynb (286 lines of code) (raw):