sdk/python/foundation-models/nvidia-nim-llama3-8b/webrequests.ipynb (331 lines of code) (raw):
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Use Azure API with Meta Llama 3.1 - 8b Instruct NIM in Azure AI Foundry and Azure ML\n",
"\n",
"This notebook shows examples of how to use Meta Llama 3 APIs offered by Microsoft Azure. We will cover: \n",
"* HTTP requests API usage for Meta Llama 3 8b pretrained and chat models in CLI\n",
"* HTTP requests API usage for Meta Llama 3 8b pretrained and chat models in Python\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"Before we start, there are certain steps we need to take to deploy the models:\n",
"\n",
"* Register for a valid Azure account with subscription \n",
"* Make sure you have access to [Azure AI Studio](https://learn.microsoft.com/en-us/azure/ai-studio/what-is-ai-studio?tabs=home)\n",
"* Create a project and resource group\n",
"* Select Nvidia NIM: Meta Llama 3.1 -8b Instruct NIM models from Model catalog\n",
"\n",
"\n",
"\n",
"\n",
"Once deployed successfully, you should be assigned for an API endpoint and a security key for inference. \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## HTTP Requests API Usage in CLI\n",
"\n",
"### Basics\n",
"\n",
"For using the REST API, You will need to have a Endpoint url and Authentication Key associated with that endpoint. \n",
"This can be acquired from previous steps. \n",
"\n",
"In this text completion example for 8B pre-trained model, we use a simple curl call for illustration. There are three major components: \n",
"\n",
"* The `host-url` is your endpoint url with completion schema. \n",
"* The `headers` defines the content type as well as your api key. \n",
"* The `payload` or `data`, which is your prompt detail and model hyper parameters."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!curl -X POST -L https://<endpoint>.<region>.inference.ml.azure.com/v1/completions -H 'Content-Type: application/json' -H 'Authorization: your-auth-key' -d '{\"prompt\": \"Math is a\", \"max_tokens\": 30, \"temperature\": 0.7}'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For chat completion, the API schema and request payload are slightly different.\n",
"\n",
"For `host-url` the path changed to `/v1/chat/completions` and the request payload also changed to include roles in conversations. Here is a sample payload: \n",
"\n",
"```\n",
"{ \n",
" \"messages\": [ \n",
" { \n",
" \"content\": \"You are a helpful assistant.\", \n",
" \"role\": \"system\" \n",
"}, \n",
" { \n",
" \"content\": \"Hello!\", \n",
" \"role\": \"user\" \n",
" } \n",
" ], \n",
" \"max_tokens\": 50, \n",
"} \n",
"```\n",
"\n",
"Here is a sample curl call for chat completion"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!curl -X POST -L https://<endpoint>.<region>.inference.ml.azure.com/v1/chat/completions -H 'Content-Type: application/json' -H 'Authorization: your-auth-key' -d '{\"messages\":[{\"content\":\"You are a helpful assistant.\",\"role\":\"system\"},{\"content\":\"What is good about Wuhan?\",\"role\":\"user\"}], \"max_tokens\": 50}'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you compare the generation result for both text and chat completion API calls, you will notice that: \n",
"\n",
"* Text completion returns a list of `choices` for the input prompt, each contains generated text and completion information such as `logprobs`.\n",
"* Chat completion returns a list of `cnoices` each has a `message` object with completion result and using the same `message` object in the request. \n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Streaming\n",
"\n",
"One fantastic feature the API offered is the streaming capability. Streaming allows the generated tokens to be sent as data-only server-sent events whenever they become available. This is extremely important for interactive applications such as chatbots, so the user is always engaged. \n",
"\n",
"To use streaming, simply set `\"stream\":\"True\"` as part of the request payload. \n",
"In the streaming mode, the REST API response will be different from non-streaming mode.\n",
"\n",
"Here is an example: "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!curl -X POST -L https://<endpoint>.<region>.inference.ml.azure.com/v1/chat/completions -H 'Content-Type: application/json' -H 'Authorization: your-auth-key' -d '{\"messages\":[{\"content\":\"You are a helpful assistant.\",\"role\":\"system\"},{\"content\":\"What is good about Wuhan?\",\"role\":\"user\"}], \"max_tokens\": 500, \"stream\": \"True\"}'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see the result comes back as a stream of `data` objects, each contains generated information including a `choice`. \n",
"The stream terminated by a `data:[DONE]\\n\\n` message."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## HTTP Requests API Usage in Python\n",
"\n",
"Besides calling the API directly from command line tools. You can also programatically call them in Python. \n",
"\n",
"Here is an example for text completion model:\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"import json\n",
"from urllib.parse import urljoin\n",
"\n",
"base_url = \"https://<endpoint>.<region>.inference.ml.azure.com/\"\n",
"\n",
"token = \"key\"\n",
"model_name = \"meta/llama-3.1-8b-instruct\"\n",
"url = urljoin(base_url, \"v1/chat/completions\")\n",
"headers = {\n",
" \"accept\": \"application/json\",\n",
" \"Authorization\": f\"Bearer {token}\", # modify this token\n",
" \"Content-Type\": \"application/json\",\n",
"}\n",
"data = {\n",
" \"messages\": [\n",
" {\n",
" \"content\": \"You are a polite and respectful chatbot helping people plan a vacation.\",\n",
" \"role\": \"system\",\n",
" },\n",
" {\"content\": \"What should I do for a 4 day vacation in Spain?\", \"role\": \"user\"},\n",
" ],\n",
" \"model\": model_name,\n",
" \"max_tokens\": 500,\n",
" \"top_p\": 1,\n",
" \"n\": 1,\n",
" \"stream\": False,\n",
" # \"stop\": \"\\n\",\n",
" \"frequency_penalty\": 0.0,\n",
"}\n",
"\n",
"response = requests.post(url, headers=headers, json=data)\n",
"# Pretty print the JSON response\n",
"print(json.dumps(response.json(), indent=4))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Chat completion in Python is very similar, here is a quick example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import urllib.request\n",
"import json\n",
"\n",
"# Configure payload data sending to API endpoint\n",
"data = {\n",
" \"messages\": [\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"What is good about Wuhan?\"},\n",
" ],\n",
" \"max_tokens\": 500,\n",
" \"temperature\": 0.9,\n",
" \"stream\": \"False\",\n",
"}\n",
"\n",
"body = str.encode(json.dumps(data))\n",
"\n",
"# Replace the url with your API endpoint\n",
"url = \"https://your-endpoint.inference.ml.azure.com/v1/chat/completions\"\n",
"\n",
"# Replace this with the key for the endpoint\n",
"api_key = \"your-auth-key\"\n",
"\n",
"if not api_key:\n",
" raise Exception(\"API Key is missing\")\n",
"\n",
"headers = {\n",
" \"accept\": \"application/json\",\n",
" \"Authorization\": f\"Bearer {api_key}\", # modify this token\n",
" \"Content-Type\": \"application/json\",\n",
"}\n",
"req = urllib.request.Request(url, body, headers)\n",
"\n",
"try:\n",
" response = urllib.request.urlopen(req)\n",
" result = response.read()\n",
" print(result)\n",
"except urllib.error.HTTPError as error:\n",
" print(\"The request failed with status code: \" + str(error.code))\n",
" # Print the headers - they include the requert ID and the timestamp, which are useful for debugging the failure\n",
" print(error.info())\n",
" print(error.read().decode(\"utf8\", \"ignore\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However in this example, the streamed data content returns back as a single payload. It didn't stream as a serial of data events as we wished. To build true streaming capabilities utilizing the API endpoint, we will utilize [`requests`](https://requests.readthedocs.io/en/latest/) library instead."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Streaming in Python\n",
"\n",
"`Requests` library is a simple HTTP library for Python built with [`urllib3`](https://github.com/urllib3/urllib3). It automatically maintains the keep-alive and HTTP connection pooling. With the `Session` class, we can easily stream the result from our API calls. \n",
"\n",
"Here is a quick example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import requests\n",
"\n",
"data = {\n",
" \"messages\": [\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"What is good about Wuhan?\"},\n",
" ],\n",
" \"max_tokens\": 500,\n",
" \"temperature\": 0.9,\n",
" \"stream\": \"True\",\n",
"}\n",
"\n",
"\n",
"def post_stream(url):\n",
" s = requests.Session()\n",
" api_key = \"key\"\n",
" # headers = {\"Content-Type\": \"application/json\", \"Authorization\": (api_key)}\n",
" headers = {\n",
" \"accept\": \"application/json\",\n",
" \"Authorization\": f\"Bearer {api_key}\", # modify this token\n",
" \"Content-Type\": \"application/json\",\n",
" }\n",
" with s.post(url, data=json.dumps(data), headers=headers, stream=True) as resp:\n",
" print(resp.status_code)\n",
" for line in resp.iter_lines():\n",
" if line:\n",
" print(line)\n",
"\n",
"\n",
"url = \"https://<endpoint>.<region>.inference.ml.azure.com/v1/chat/completions\"\n",
"post_stream(url)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "dev",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.16"
}
},
"nbformat": 4,
"nbformat_minor": 2
}