sdk/python/foundation-models/nvidia-nim-llama3-8b/aiinference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use AI Inference SDK with Meta Llama 3.1 - 8b Instruct NIM in Azure AI Foundry and Azure ML\n",
    "\n",
    "This demo notebook shows how to consume Meta-llama-3.1-8B NIM deployments in Azure AI Foundry and Azure AML using AI Inference SDK with Meta Llama 3 NIM "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "Before we start, there are certain steps we need to take to deploy the models:\n",
    "\n",
    "* Register for a valid Azure account with subscription \n",
    "* Make sure you have access to [Azure AI Studio](https://learn.microsoft.com/en-us/azure/ai-studio/what-is-ai-studio?tabs=home)\n",
    "* Create a project and resource group\n",
    "* Select Nvidia NIM:  Meta Llama 3.1 -8b Instruct NIM models from Model catalog\n",
    "\n",
    "![nim-models.png](nim-models.png)\n",
    "\n",
    "Once deployed successfully, you should be assigned for an API endpoint and a security key for inference. \n",
    "\n",
    "\n",
    "Install the package azure-ai-inference using your package manager, like pip:\n",
    "\n",
    "```\n",
    "pip install azure-ai-inference\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.inference import ChatCompletionsClient\n",
    "from azure.core.credentials import AzureKeyCredential\n",
    "from azure.ai.inference.models import SystemMessage, UserMessage\n",
    "\n",
    "endpoint = \"https://<endpoint>.<region>.inference.ml.azure.com/v1\"\n",
    "key = os.getenv(\"AZURE_AI_CHAT_KEY\", \"keyhere\")\n",
    "\n",
    "client = ChatCompletionsClient(\n",
    "    endpoint=endpoint,\n",
    "    credential=AzureKeyCredential(key),\n",
    "    # model=model\n",
    ")\n",
    "\n",
    "response = client.complete(\n",
    "    messages=[\n",
    "        SystemMessage(\"You are a helpful assistant.\"),\n",
    "        UserMessage(\"Can you write me a song?\"),\n",
    "    ],\n",
    ")\n",
    "\n",
    "print(response.choices[0].message.content)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}