Lab3_risk_safety_eval/safety_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Risk and safety evaluator usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "When utilizing AI-assisted risk and safety metrics, a GPT model is not necessary. Instead of model_config, provide your azure_ai_project information. This will access the Azure AI project safety evaluations back-end service, which provisions a GPT model specifically for harm evaluation, generating content risk severity scores and reasoning for the safety evaluators.\n",
    "\n",
    "Safety evaluations include Hate and unfairness, Sexual, Violent, Self-harm, Indirect attack\t\n",
    "\n",
    "## Region Support\n",
    "Currently, AI-assisted risk and safety metrics are available only in the following regions at time of writing:\n",
    "https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/evaluate-sdk\n",
    "\n",
    "- East US 2\n",
    "- Sweden Central\n",
    "- France Central\n",
    "- Switzerland West"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#%pip install azure-ai-evaluation\n",
    "#%pip install azure.identity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pprint import pprint\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"../.credentials2.env\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Azure AI project and Azure OpenAI conncetion with your environment variables\n",
    "azure_ai_project = {\n",
    "    \"subscription_id\": os.environ.get(\"AZURE_SUBSCRIPTION_ID\"),\n",
    "    \"resource_group_name\": os.environ.get(\"AZURE_RESOURCE_GROUP\"),\n",
    "    \"project_name\": os.environ.get(\"AZURE_PROJECT_NAME\"),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.evaluation import ViolenceEvaluator\n",
    "from azure.identity import DefaultAzureCredential\n",
    "credential = DefaultAzureCredential()\n",
    "\n",
    "\n",
    "# Initializing Violence Evaluator with project information\n",
    "violence_eval = ViolenceEvaluator(credential=credential, azure_ai_project=azure_ai_project)\n",
    "# Running Violence Evaluator on a query and response pair\n",
    "violence_score = violence_eval(query=\"What is the capital of United Kingdom?\", response=\"London.\")\n",
    "print(violence_score)\n",
    "\n",
    "# Conversation mode\n",
    "import json\n",
    "\n",
    "conversation_str =  \"\"\"{\"messages\": [ { \"content\": \"Which tent is the most waterproof?\", \"role\": \"user\" }, { \"content\": \"The Alpine Explorer Tent is the most waterproof\", \"role\": \"assistant\", \"context\": \"From the our product list the alpine explorer tent is the most waterproof. The Adventure Dining Table has higher weight.\" }, { \"content\": \"How much does it cost?\", \"role\": \"user\" }, { \"content\": \"$120.\", \"role\": \"assistant\", \"context\": \"The Alpine Explorer Tent is $120.\"} ] }\"\"\" \n",
    "conversation = json.loads(conversation_str)\n",
    "\n",
    "violence_conv_score = violence_eval(conversation=conversation) \n",
    "\n",
    "print(violence_conv_score)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "model_eval",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}