7-reasoning/flightCancellationPolicy/evalsAndPromptImprovements.ipynb (6,338 lines of code) (raw):

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Evals and Metaprompting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Initial Setup\n", "We'll start with importing a few dependencies and setting up some visuals" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%pip install openai\n", "%pip install pandas\n", "%pip install ipython" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from openai import OpenAI\n", "from IPython.display import display, Markdown\n", "import pandas as pd\n", "from concurrent.futures import ThreadPoolExecutor\n", "from functionDefinitions import TOOLS\n", "import csv\n", "import json\n", "import os\n", "\n", "client = OpenAI()\n", "MODEL = 'o1-mini'\n", "\n", "pd.set_option('display.max_colwidth', None)\n", "pd.set_option('display.max_rows', None)\n", "pd.set_option('display.max_columns', None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generating a Routine\n", "We'll take the Flight Cancellation Policy that we have created and convert it to an LLM-based routine with the following prompt" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "with open('originalPolicy/flightCancellationsPolicy.md', 'r') as file:\n", " flight_cancellation_policy = file.read()" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "CONVERSION_PROMPT = \"\"\"\n", "You are a helpful assistant tasked with taking an external facing help center article and converting it into a internal-facing programmatically executable routine optimized for an LLM. \n", "The LLM using this routine will be tasked with reading the policy, answering incoming questions from customers, and helping drive the case toward resolution.\n", "\n", "Please follow these instructions:\n", "1. **Review the customer service policy carefully** to ensure every step is accounted for. It is crucial not to skip any steps or policies.\n", "2. **Organize the instructions into a logical, step-by-step order**, using the specified format.\n", "3. **Use the following format**:\n", " - **Main actions are numbered** (e.g., 1, 2, 3).\n", " - **Sub-actions are lettered** under their relevant main actions (e.g., 1a, 1b).\n", " **Sub-actions should start on new lines**\n", " - **Specify conditions using clear 'if...then...else' statements** (e.g., 'If the product was purchased within 30 days, then...').\n", " - **For instructions that require more information from the customer**, provide polite and professional prompts to ask for additional information.\n", " - **For actions that require data from external systems**, write a step to call a function using backticks for the function name (e.g., `call the check_delivery_date function`).\n", " - **If a step requires the customer service agent to take an action** (e.g., process a refund), generate a function call for this action (e.g., `call the process_refund function`).\n", " - **Define any new functions** by providing a brief description of their purpose and required parameters.\n", " - **If there is an action an assistant can performon behalf of the user**, include a function call for this action (e.g., `call the change_email_address function`), and ensure the function is defined with its purpose and required parameters.\n", " - This action may not be explicitly defined in the help center article, but can be done to help the user resolve their inquiry faster\n", " - **The step prior to case resolution should always be to ask if there is anything more you can assist with**.\n", " - **End with a final action for case resolution**: calling the `case_resolution` function should always be the final step.\n", "4. **Ensure compliance** by making sure all steps adhere to company policies, privacy regulations, and legal requirements.\n", "5. **Handle exceptions or escalations** by specifying steps for scenarios that fall outside the standard policy.\n", "\n", "**Important**: If at any point you are uncertain, respond with \"I don't know.\"\n", "\n", "Please convert the customer service policy into the formatted routine, ensuring it is easy to follow and execute programmatically.\n", "\n", "\"\"\"" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def generate_routine(policy):\n", " try:\n", " messages = [\n", " {\n", " \"role\": \"user\",\n", " \"content\": f\"\"\"\n", " {CONVERSION_PROMPT}\n", "\n", " POLICY:\n", " {policy}\n", " \"\"\"\n", " }\n", " ]\n", "\n", " response = client.chat.completions.create(\n", " model=MODEL,\n", " messages=messages\n", " )\n", " \n", "\n", " return response.choices[0].message.content \n", " except Exception as e:\n", " print(f\"An error occurred: {e}\")" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "flight_cancellation_routine = generate_routine(flight_cancellation_policy)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "```markdown\n", "1. **Confirm Customer Identity**\n", " a. Prompt the customer: \"Could you please provide your booking reference, full name, and flight number?\"\n", " b. `call the verify_identity function` with parameters booking_reference, full_name, flight_number\n", " - **verify_identity**: Validates the customer's identity based on booking reference, name, and flight number.\n", "\n", "2. **Listen and Understand Customer Request**\n", " a. Ask the customer: \"Are you looking to cancel your flight, make a change, or inquire about compensation?\"\n", " b. If the customer wants to **cancel**, then proceed to step 3.\n", " c. Else if the customer wants to **change**, then proceed to step 4.\n", " d. Else if the customer inquires about **compensation**, then proceed to step 5.\n", " e. Else, prompt: \"Could you please clarify your request?\" and loop back to step 2a.\n", "\n", "3. **Handle Cancellations**\n", " a. `call the check_ticket_type function` with parameter booking_reference\n", " - **check_ticket_type**: Determines if the ticket is refundable, non-refundable, or flexible.\n", " b. If the ticket is **refundable**, then:\n", " i. `call the process_refund function` with parameters booking_reference, refund_amount\n", " - **process_refund**: Initiates a refund based on the refund amount and timing.\n", " ii. Inform the customer: \"Your refundable ticket qualifies for a full or partial refund based on the cancellation timing.\"\n", " - If **within 24 hours of booking**, then full refund is guaranteed.\n", " - Else, provide refund amount based on specific fare rules.\n", " c. Else if the ticket is **non-refundable**, then:\n", " i. Offer flight credit: \"We can provide you with flight credit for future use, subject to an administration fee.\"\n", " - `call the offer_flight_credit function` with parameters booking_reference, administration_fee\n", " - **offer_flight_credit**: Issues flight credit after deducting any applicable fees.\n", " ii. Advise about penalty fees: \"Please note that a penalty fee of [amount] applies to non-refundable tickets.\"\n", " - `call the apply_penalty_fee function` with parameters booking_reference, penalty_amount\n", " - **apply_penalty_fee**: Deducts the penalty fee from the available refund or credit.\n", " d. Else, inform the customer: \"I don't know.\"\n", "\n", "4. **Handle Changes**\n", " a. `call the check_ticket_type function` with parameter booking_reference\n", " - **check_ticket_type**: Determines if the ticket is refundable, non-refundable, or flexible.\n", " b. Ask the customer: \"Would you like to make a same-day change or a change in advance?\"\n", " c. If **same-day change**, then:\n", " i. If the ticket is **flexible**, then:\n", " - `call the process_same_day_change function` with parameters booking_reference, new_flight_details\n", " - **process_same_day_change**: Changes the flight without any fees, subject to availability.\n", " ii. Else if the ticket is **non-flexible**, then:\n", " - `call the apply_change_fee function` with parameters booking_reference, change_fee\n", " - **apply_change_fee**: Applies a change fee to the ticket.\n", " - Ask the customer: \"There is a change fee of [amount]. Would you like to proceed and pay any fare difference?\"\n", " - If yes, `call the process_fare_difference function` with parameters booking_reference, fare_difference\n", " - **process_fare_difference**: Calculates and processes any additional fare required.\n", " d. Else if **change in advance**, then:\n", " i. Ask: \"Is your requested change within 7 days of departure?\"\n", " ii. If **within 7 days**, then:\n", " - `call the apply_standard_change_fee function` with parameters booking_reference, standard_change_fee\n", " - **apply_standard_change_fee**: Applies the standard change fee for changes within 7 days.\n", " iii. Else (**beyond 7 days**), then:\n", " - If the ticket type allows, `call the apply_lesser_change_fee function` or `call the waive_change_fee function` based on ticket flexibility\n", " - **apply_lesser_change_fee**: Applies a reduced change fee.\n", " - **waive_change_fee**: Removes the change fee for eligible tickets.\n", " e. Else, inform the customer: \"I don't know.\"\n", "\n", "5. **Handle Compensation Inquiries**\n", " a. `call the check_compensation_eligibility function` with parameters booking_reference, delay_length, cause\n", " - **check_compensation_eligibility**: Determines if the customer is eligible for compensation based on delay and cause.\n", " b. If **eligible for compensation**, then:\n", " i. Inform the customer: \"You are eligible for compensation based on the length of your delay and its cause.\"\n", " ii. If the **delay exceeds four hours**, then:\n", " - `call the provide_meal_voucher function` or `call the offer_hotel_accommodation function` based on customer needs\n", " - **provide_meal_voucher**: Issues meal vouchers to the customer.\n", " - **offer_hotel_accommodation**: Arranges hotel accommodation if an overnight stay is required.\n", " c. Else, inform the customer: \"I don't know.\"\n", "\n", "6. **Rebooking Process**\n", " a. `call the get_next_available_flight function` with parameter booking_reference\n", " - **get_next_available_flight**: Retrieves the next available flight operated by the airline.\n", " b. If a suitable flight is available, then:\n", " i. `call the process_rebooking function` with parameters booking_reference, new_flight_details\n", " - **process_rebooking**: Rebooks the customer on the selected flight.\n", " ii. Inform the customer: \"You have been rebooked on the next available flight.\"\n", " c. Else, if no suitable flight is available, then:\n", " i. `call the check_interline_partners function` with parameter booking_reference\n", " - **check_interline_partners**: Checks for alternative connections with partner airlines.\n", " ii. If an interline option is found, then:\n", " - `call the process_interline_rebooking function` with parameters booking_reference, partner_flight_details\n", " - **process_interline_rebooking**: Rebooks the customer with a partner airline.\n", " - Inform the customer: \"Your flight has been rebooked with our partner airline.\"\n", " iii. Else, inform the customer: \"Currently, there are no available flights. Would you like to be placed on a waitlist or receive an upgrade if possible?\"\n", " - If the customer opts for an upgrade, `call the offer_upgraded_seat function` with parameters booking_reference\n", " - **offer_upgraded_seat**: Offers a complimentary upgrade to a higher class if available.\n", " d. Else, inform the customer: \"I don't know.\"\n", "\n", "7. **Special Cases Handling**\n", " a. If the customer cites a **medical emergency**, then:\n", " i. Ask for documentation: \"Could you please provide a medical certificate?\"\n", " ii. If provided, `call the process_medical_cancellation function` with parameters booking_reference, medical_certificate\n", " - **process_medical_cancellation**: Allows full cancellation or flight credit without fees.\n", " b. If the customer cites a **bereavement**, then:\n", " i. Ask for documentation: \"Could you please provide the necessary documentation for bereavement?\"\n", " ii. If provided, `call the process_bereavement_case function` with parameters booking_reference, documentation\n", " - **process_bereavement_case**: Offers flexibility on cancellations or changes.\n", " c. If the booking is a **group booking**, then:\n", " i. Ask the customer: \"Is this a group booking?\"\n", " ii. If yes, `call the handle_group_booking function` with parameters booking_reference, group_request\n", " - **handle_group_booking**: Manages partial cancellations or name changes within the group.\n", " d. If the booking involves **unaccompanied minors**, then:\n", " i. `call the prioritize_minors function` with parameters booking_reference\n", " - **prioritize_minors**: Ensures unaccompanied minors are rebooked on the next available flight with proper supervision.\n", " e. Else, proceed to next step.\n", "\n", "8. **Ask for Additional Assistance**\n", " a. Prompt the customer: \"Is there anything else I can assist you with today?\"\n", " b. If the customer has additional requests, loop back to step 2.\n", " c. Else, proceed to step 9.\n", "\n", "9. **Case Resolution**\n", " a. `call the case_resolution function` with parameters booking_reference, final_status\n", " - **case_resolution**: Marks the case as resolved and logs the final status.\n", "\n", "10. **Handle Exceptions or Escalations**\n", " a. If at any point the issue falls outside standard policies, then:\n", " i. Inform the customer: \"I understand your concern. Let me escalate this to a supervisor for further assistance.\"\n", " ii. `call the escalate_to_supervisor function` with parameters booking_reference, issue_details\n", " - **escalate_to_supervisor**: Forwards the case to a supervisor for specialized handling.\n", " b. Else, inform the customer: \"I don't know.\"\n", "```" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "display(Markdown(flight_cancellation_routine))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluating Accuracy\n", "\n", "Now that we have a routine generated with o1, we can run it against our evaluation suite and measure its accuracy.\n", "\n", "We'll start by creating an agent that is equipped with the policy and a list of tools. It will be given messages from an existing conversation and will be tasked with determining the next best action to take" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def agent_response(transcript, policy, model):\n", " try:\n", " messages = [\n", " {\n", " \"role\": \"system\",\n", " \"content\": f\"\"\"\n", "You are a customer service agent that is responsible for handling airline related issues. Below is the exact policy that you must follow to address the customer's issue\n", "\n", "POLICY:\n", "{policy}\n", " \"\"\"\n", " }\n", " ]\n", "\n", " messages.extend(transcript)\n", " response = client.chat.completions.create(\n", " model=model,\n", " messages=messages,\n", " tools=TOOLS\n", " )\n", " \n", " return response.choices[0].message \n", " except Exception as e:\n", " print(f\"An error occurred: {e}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will process each row in parallel to reduce runtime and compare the function call + inputs that the model selects against our expected function + parameters." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "def process_row(row_number, row, policy, model):\n", " try:\n", " # Extract values from the current row\n", " conversation_str = row['conversation']\n", " expected_function = row['expected_function']\n", " expected_inputs_str = row['expected_inputs']\n", "\n", " # Parse the conversation JSON\n", " try:\n", " conversation = json.loads(conversation_str)\n", " except json.JSONDecodeError as e:\n", " print(f\"Error parsing 'conversation' in row {row_number}: {e}\")\n", " conversation = None\n", "\n", " # Parse the expected_inputs JSON\n", " try:\n", " expected_inputs = json.loads(expected_inputs_str)\n", " # If expected_inputs is a string (double-encoded), parse it again\n", " if isinstance(expected_inputs, str):\n", " expected_inputs = json.loads(expected_inputs)\n", " except json.JSONDecodeError as e:\n", " print(f\"Error parsing 'expected_inputs' in row {row_number}: {e}\")\n", " expected_inputs = None\n", "\n", " # Extract the last assistant's message content if it exists\n", " response = agent_response(conversation, policy, model)\n", " assistant_message_content = response.content if response else None\n", " tool_calls = response.tool_calls\n", "\n", " # If the most recent response does not contain a tool call and just a message from the assistant, we rerun it once more to get our tool call.\n", " if not tool_calls:\n", " if assistant_message_content:\n", " # Append the assistant's message content to the conversation\n", " conversation.append({\"role\": \"assistant\", \"content\": assistant_message_content})\n", " # Make another call to agent_response\n", " response = agent_response(conversation, policy, model)\n", " tool_calls = response.tool_calls\n", "\n", " if not tool_calls:\n", " actual_function = None\n", " actual_inputs = None\n", " is_correct = False\n", " else:\n", " tool_call = tool_calls[0] # Assuming we're only interested in the first tool call\n", " function_name = tool_call.function.name\n", " arguments = json.loads(tool_call.function.arguments)\n", "\n", " is_correct = (function_name == expected_function) and (arguments == expected_inputs)\n", " actual_function = function_name\n", " actual_inputs = arguments\n", "\n", " return {\n", " 'expected_function': expected_function,\n", " 'expected_inputs': expected_inputs,\n", " 'actual_function': actual_function,\n", " 'actual_inputs': actual_inputs,\n", " 'is_correct': is_correct,\n", " 'assistant_message_content': assistant_message_content\n", " }\n", "\n", " except Exception as e:\n", " print(f\"Error processing row {row_number}: {e}\")\n", " return {\n", " 'expected_function': row.get('expected_function'),\n", " 'expected_inputs': row.get('expected_inputs'),\n", " 'actual_function': None,\n", " 'actual_inputs': None,\n", " 'is_correct': False,\n", " 'assistant_message_content': None\n", " }\n", "\n", "def evaluate_function_calls(file_path, policy, model):\n", " records = []\n", "\n", " # Check if the file exists\n", " if not os.path.isfile(file_path):\n", " print(f\"Error: The file '{file_path}' does not exist.\")\n", " return\n", "\n", " try:\n", " with open(file_path, 'r', newline='', encoding='utf-8') as csvfile:\n", " # Initialize the CSV reader with pipe as delimiter\n", " reader = csv.DictReader(csvfile, delimiter='|', quotechar='\"', escapechar='\\\\')\n", "\n", " # Use ThreadPoolExecutor to process rows in parallel\n", " with ThreadPoolExecutor() as executor:\n", " futures = {executor.submit(process_row, row_number, row, policy, model): row_number for row_number, row in enumerate(reader, start=2)}\n", " for future in futures:\n", " record = future.result()\n", " records.append(record)\n", "\n", " except Exception as e:\n", " print(f\"An unexpected error occurred while reading the CSV file: {e}\")\n", " return\n", "\n", " df = pd.DataFrame(records)\n", " total_accuracy = df['is_correct'].mean()\n", " return df, total_accuracy\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's take a look at our results." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "### Accuracy: 70.59%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>You have a non-refundable ticket. I can provide you with flight credit for future use, subject to an administration fee. Additionally, please note that a penalty fee applies to non-refundable tickets.\\n\\nWould you like me to proceed with offering you flight credit, or do you need any further information?</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>None</td>\n", " <td>None</td>\n", " <td>False</td>\n", " <td>Would you like to make a same-day change or a change in advance?</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>process_partial_group_cancellation</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_next_available_flight</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>check_next_available_flight</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>False</td>\n", " <td>Are you looking to cancel your flight, make a change, or inquire about compensation?</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 check_ticket_type \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 None \n", "8 process_flexible_cancellation \n", "9 process_partial_group_cancellation \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 check_next_available_flight \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 check_next_available_flight \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 None \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 False \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 False \n", "8 True \n", "9 False \n", "10 True \n", "11 True \n", "12 True \n", "13 False \n", "14 True \n", "15 True \n", "16 False \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 You have a non-refundable ticket. I can provide you with flight credit for future use, subject to an administration fee. Additionally, please note that a penalty fee applies to non-refundable tickets.\\n\\nWould you like me to proceed with offering you flight credit, or do you need any further information? \n", "4 None \n", "5 None \n", "6 None \n", "7 Would you like to make a same-day change or a change in advance? \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 Are you looking to cancel your flight, make a change, or inquire about compensation? " ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Assuming the CSV file is located at 'evals/functionCallingEval.csv'\n", "df, accuracy = evaluate_function_calls('evals/functionCallingEval.csv', flight_cancellation_routine, 'gpt-4o-mini-2024-07-18')\n", "\n", "# Display the accuracy as a mini header\n", "display(Markdown(f\"### Accuracy: {accuracy:.2%}\"))\n", "\n", "display(df)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Metaprompting\n", "\n", "Let's now leverage o1 again to add in a metaprompting loop to see if we can improve the quality of our evals.\n", "\n", "We'll take the following multi-step approach:\n", "- We'll pass in the current routine + eval results to o1 and ask it analyze the results and update the routine accordingly\n", "- Since o1 does not currently support structured outputs, we'll chain with output with a 4o to enforce a schema we can parse\n", "- Finally, we take the new routine and run it back through our eval to generate new results\n", "\n", "We'll run this loop a fixed number of times and see what improvements we can make" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "def metaprompt(messages):\n", " try:\n", " response = client.chat.completions.create(\n", " model='o1-preview',\n", " messages=messages,\n", " )\n", " \n", " return response.choices[0].message.content\n", " except Exception as e:\n", " print(f\"An error occurred: {e}\")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "def enforce_schema(updated_prompt):\n", " try:\n", " messages = [\n", " {\n", " \"role\": \"system\",\n", " \"content\": f\"\"\"\n", "You will be given a response from an LLM that just generated a policy for flight cancellations.\n", "Your task is to take just the policy exactly as it is written and return it in the defined json. Remove all parts from the LLM's answer that are not part of the policy.\n", "\n", "LLM RESPONSE:\n", "{updated_prompt}\n", " \"\"\"\n", " }\n", " ]\n", "\n", " response = client.chat.completions.create(\n", " model='gpt-4o-2024-08-06',\n", " messages=messages,\n", " response_format= {\n", " \"type\": \"json_schema\",\n", " \"json_schema\": {\n", " \"name\": \"policy_output\",\n", " \"schema\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"final_answer\": { \"type\": \"string\" }\n", " },\n", " \"required\": [\"final_answer\"],\n", " \"additionalProperties\": False\n", " },\n", " \"strict\": True\n", " }\n", " }\n", "\n", " )\n", " \n", " return response.choices[0].message.content\n", " except Exception as e:\n", " print(f\"An error occurred: {e}\") " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "### Accuracy: 64.71%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>None</td>\n", " <td>None</td>\n", " <td>False</td>\n", " <td>Since you have a non-refundable ticket, we can offer you flight credit for future use, subject to an administration fee. Please note that a penalty fee will also apply to non-refundable tickets.\\n\\nWould you like to proceed with the flight credit option?</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>None</td>\n", " <td>None</td>\n", " <td>False</td>\n", " <td>Would you like to make a same-day change or a change in advance?</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>Please provide me with the new flight details that you would like to change to.</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_next_available_flight</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>check_next_available_flight</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>False</td>\n", " <td>Are you looking to be rebooked on the next available flight?</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 check_ticket_type \n", "3 None \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 None \n", "8 process_flexible_cancellation \n", "9 check_ticket_type \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 check_next_available_flight \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 check_next_available_flight \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 None \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 None \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 False \n", "3 False \n", "4 True \n", "5 True \n", "6 True \n", "7 False \n", "8 True \n", "9 False \n", "10 True \n", "11 True \n", "12 True \n", "13 False \n", "14 True \n", "15 True \n", "16 False \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 Since you have a non-refundable ticket, we can offer you flight credit for future use, subject to an administration fee. Please note that a penalty fee will also apply to non-refundable tickets.\\n\\nWould you like to proceed with the flight credit option? \n", "4 None \n", "5 None \n", "6 None \n", "7 Would you like to make a same-day change or a change in advance? \n", "8 None \n", "9 None \n", "10 None \n", "11 Please provide me with the new flight details that you would like to change to. \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 Are you looking to be rebooked on the next available flight? " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: **Internal Flight Cancellations and Changes Policy**\n", "\n", "**Purpose**: This document serves as a detailed guide for internal support agents to handle flight cancellations and changes. The focus is on providing clear instructions, ensuring efficiency, consistency, and customer satisfaction.\n", "\n", "**Note**: Always maintain a calm, empathetic tone while assisting customers, especially during stressful situations involving cancellations or major changes.\n", "\n", "## **Table of Contents**\n", "\n", "1. General Guidelines for Handling Customer Requests \n", "2. Functions and Procedures \n", "3. Cancellations: Types and Policies \n", "4. Changes: Types and Policies \n", "5. Rebooking Guidelines \n", "6. Compensation and Refund Rules \n", "7. Special Cases \n", "8. FAQs for Common Scenarios\n", "\n", "---\n", "\n", "### **1\\. General Guidelines for Handling Customer Requests**\n", "\n", "- **Confirm Identity**: Verify the customer's identity by asking for their booking reference and any additional required details (e.g., name and flight number).\n", " \n", "- **Listen and Understand**: Determine if the customer wants to cancel, change, or inquire about compensation.\n", " \n", "- **Use Appropriate Functions**: Directly call the relevant function to process the customer's request, based on the policy guidelines.\n", " \n", "- **Minimal Dialogue**: Avoid unnecessary messages or questions. Proceed with the required action unless additional information is needed.\n", " \n", "- **Be Empathetic**: Express empathy, especially when the customer is experiencing inconvenience due to the airline.\n", "\n", "---\n", "\n", "### **2\\. Functions and Procedures**\n", "\n", "To ensure consistency and efficiency, use the following predefined functions when assisting customers:\n", "\n", "- **check_ticket_type(booking_reference)**: Determines the ticket type (non-refundable, refundable, or flexible).\n", " \n", "- **process_full_refund(booking_reference)**: Processes a full refund for the customer.\n", " \n", "- **offer_flight_credit(booking_reference)**: Offers flight credit for future use.\n", " \n", "- **apply_change_fee(booking_reference)**: Applies any applicable change fees to the booking.\n", " \n", "- **process_change_no_fee(booking_reference)**: Processes a change to the booking without any fees.\n", " \n", "- **rebook_without_fee(booking_reference)**: Rebooks the customer on the next available flight without additional charges.\n", " \n", "- **offer_accommodation(booking_reference)**: Arranges hotel accommodation for the customer.\n", " \n", "- **check_next_available_flight(booking_reference)**: Checks for the next available flight options.\n", " \n", "- **prioritize_missed_connections(booking_reference)**: Prioritizes rebooking for missed connections due to airline delays.\n", " \n", "- **check_compensation_eligibility(booking_reference)**: Checks if the customer is eligible for compensation.\n", " \n", "- **permit_name_change(booking_reference)**: Processes a name change on the booking.\n", " \n", "- **process_flexible_cancellation(booking_reference, documentation)**: Processes cancellations with flexibility due to special circumstances (e.g., medical emergencies).\n", "\n", "**Function Calling Guidelines**:\n", "\n", "- **Direct Action**: When a policy dictates a specific action, call the corresponding function directly without unnecessary conversation.\n", " \n", "- **Information Gathering**: Only ask for additional information if it is essential to proceed.\n", " \n", "- **Efficiency**: Focus on resolving the customer's request promptly by using the appropriate function.\n", "\n", "---\n", "\n", "### **3\\. Cancellations: Types and Policies**\n", "\n", "#### **3.1 Customer-Initiated Cancellations**\n", "\n", "- **Refundable Tickets**:\n", " \n", " - **Within 24 Hours of Booking**: \n", " - **Action**: Call **process_full_refund(booking_reference)**.\n", " \n", " - **Beyond 24 Hours**:\n", " - **Action**: Call **process_full_refund(booking_reference)** if fare rules allow; otherwise, inform the customer of any restrictions before proceeding.\n", " \n", "- **Non-Refundable Tickets**:\n", " \n", " - **Flight Credit**:\n", " - **Action**: Call **offer_flight_credit(booking_reference)**.\n", " \n", " - **Penalty Fee**:\n", " - **Action**: Inform the customer of any applicable penalty fees before offering flight credit.\n", " \n", "- **Flexible Tickets**:\n", " \n", " - **Action**: Call **process_full_refund(booking_reference)** or **offer_flight_credit(booking_reference)** based on customer preference.\n", "\n", "#### **3.2 Airline-Initiated Cancellations**\n", "\n", "- **Weather-Related Cancellations**:\n", " \n", " - **Rebook Without Fees**:\n", " - **Action**: Call **rebook_without_fee(booking_reference)**.\n", " \n", " - **Refund Options**:\n", " - **Action**: If rebooking is unacceptable, call **process_full_refund(booking_reference)**.\n", " \n", "- **Mechanical Issues/Operational Changes**:\n", " \n", " - **Rebooking Priority**:\n", " - **Action**: Call **rebook_without_fee(booking_reference)** or **prioritize_missed_connections(booking_reference)**.\n", " \n", " - **Accommodation**:\n", " - **Action**: If overnight stay is required, call **offer_accommodation(booking_reference)**.\n", "\n", "#### **3.3 No-Show Policy**\n", "\n", "- **Non-Refundable Tickets**:\n", " \n", " - **Action**: If applicable, call **offer_flight_credit(booking_reference)**.\n", " \n", "- **Refundable Tickets**:\n", " \n", " - **Action**: Call **process_full_refund(booking_reference)** for any refundable taxes or fees.\n", "\n", "---\n", "\n", "### **4\\. Changes: Types and Policies**\n", "\n", "#### **4.1 Customer-Initiated Changes**\n", "\n", "- **Same-Day Changes**:\n", " \n", " - **Flexible Tickets**:\n", " - **Action**: Call **process_change_no_fee(booking_reference)**.\n", " \n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **apply_change_fee(booking_reference)**.\n", " \n", "- **Changes in Advance**:\n", " \n", " - **Within 7 Days of Departure**:\n", " - **Action**: Call **apply_change_fee(booking_reference)**.\n", " \n", " - **Beyond 7 Days**:\n", " - **Flexible Tickets**:\n", " - **Action**: Call **process_change_no_fee(booking_reference)**.\n", " \n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **apply_change_fee(booking_reference)**.\n", "\n", "#### **4.2 Airline-Initiated Changes**\n", "\n", "- **Schedule Changes**:\n", " \n", " - **Minor Changes (< 2 Hours)**:\n", " - **Action**: Inform the customer of the new schedule. Only proceed if the customer requests alternatives.\n", " \n", " - **Major Changes (≥ 2 Hours)**:\n", " - **Action**: Offer options by calling:\n", " - **rebook_without_fee(booking_reference)**\n", " - **process_full_refund(booking_reference)**\n", " - **offer_flight_credit(booking_reference)**\n", "\n", "---\n", "\n", "### **5\\. Rebooking Guidelines**\n", "\n", "- **Priority Handling**:\n", " \n", " - **Action**: Call **rebook_without_fee(booking_reference)** to rebook on the next available flight with our airline.\n", " \n", "- **Interline Agreements**:\n", " \n", " - **Action**: If no suitable options are available, check for flights with partner airlines. Proceed accordingly.\n", " \n", "- **Upgrade Availability**:\n", " \n", " - **Action**: If only higher-class seats are available, rebook without additional charge.\n", "\n", "---\n", "\n", "### **6\\. Compensation and Refund Rules**\n", "\n", "#### **6.1 Refund Processing**\n", "\n", "- **Refund Timeline**:\n", " \n", " - **Action**: Inform the customer that refunds are processed within 7-10 business days for credit cards and 14-20 business days for debit cards.\n", " \n", "- **Refund Options**:\n", " \n", " - **Original Payment Method**:\n", " - **Action**: Default to refunding the original payment method.\n", " \n", " - **Travel Credits**:\n", " - **Action**: Call **offer_flight_credit(booking_reference)** if the customer prefers.\n", "\n", "#### **6.2 Compensation**\n", "\n", "- **Eligibility Check**:\n", " \n", " - **Action**: Call **check_compensation_eligibility(booking_reference)**.\n", " \n", "- **Meals and Accommodation**:\n", " \n", " - **Action**: If eligible, call **offer_accommodation(booking_reference)**.\n", "\n", "---\n", "\n", "### **7\\. Special Cases**\n", "\n", "#### **7.1 Medical Emergencies**\n", "\n", "- **Flexible Cancellations**:\n", " \n", " - **Action**: Upon receiving a medical certificate, call **process_flexible_cancellation(booking_reference, documentation)**.\n", "\n", "#### **7.2 Bereavement Cases**\n", "\n", "- **Special Flexibility**:\n", " \n", " - **Action**: Call **process_flexible_cancellation(booking_reference, documentation)** when proper documentation is provided.\n", "\n", "#### **7.3 Group Bookings**\n", "\n", "- **Partial Cancellations**:\n", " \n", " - **Action**: Process individual cancellations as per ticket policies.\n", " \n", "- **Name Changes**:\n", " \n", " - **Action**: If requested 7+ days before departure, call **permit_name_change(booking_reference)**.\n", "\n", "#### **7.4 Unaccompanied Minors**\n", "\n", "- **Rebooking with Supervision**:\n", " \n", " - **Action**: Call **rebook_without_fee(booking_reference)**, ensuring supervision arrangements are made.\n", "\n", "- **Priority for Minors**:\n", " \n", " - **Action**: Prioritize unaccompanied minors by calling **prioritize_missed_connections(booking_reference)** if necessary.\n", "\n", "---\n", "\n", "### **8\\. FAQs for Common Scenarios**\n", "\n", "**Q1: What if a customer’s connecting flight is affected?**\n", "\n", "- **Action**: Call **prioritize_missed_connections(booking_reference)** to rebook the customer promptly.\n", "\n", "---\n", "\n", "**Q2: How do I handle customers seeking upgrades after a cancellation?**\n", "\n", "- **Action**: If available, rebook using **rebook_without_fee(booking_reference)** and inform the customer of the upgrade at no extra cost.\n", "\n", "---\n", "\n", "**Q3: What if a customer needs to change a destination?**\n", "\n", "- **Action**: Inform the customer about fare differences and any change fees. Proceed by calling **apply_change_fee(booking_reference)**.\n", "\n", "---\n", "\n", "**Q4: What are the rules around rebooking for third-party bookings?**\n", "\n", "- **Action**: Advise the customer to contact the original booking agent. If eligible, proceed with rebooking per policy.\n", "\n", "---\n", "\n", "**Q5: What should I do if the customer requests compensation for inconvenience?**\n", "\n", "- **Action**: Call **check_compensation_eligibility(booking_reference)** to assess and proceed accordingly.\n", "\n", "---\n", "\n", "**Q6: How do I handle a situation where a customer’s preferred flight is fully booked?**\n", "\n", "- **Action**: Call **check_next_available_flight(booking_reference)** to offer alternative options. If possible, place the customer on a waitlist.\n", "\n", "---\n", "\n", "**Q7: What if a customer wants to change the name on the ticket?**\n", "\n", "- **Action**: If the request is made 7+ days before departure, call **permit_name_change(booking_reference)**.\n", "\n", "---\n", "\n", "**Q8: How should I proceed if the customer is unsure about accepting flight credit?**\n", "\n", "- **Action**: Offer to process a refund by calling **process_full_refund(booking_reference)** if policy allows.\n", "\n", "---\n", "\n", "**Q9: What if the customer is a no-show due to unforeseen circumstances?**\n", "\n", "- **Action**: Assess the situation. If applicable, offer flight credit by calling **offer_flight_credit(booking_reference)**.\n", "\n", "---\n", "\n", "**Q10: How do I handle requests for same-day flight changes?**\n", "\n", "- **Action**: Determine the ticket type:\n", " - **Flexible Ticket**: Call **process_change_no_fee(booking_reference)**.\n", " - **Non-Flexible Ticket**: Call **apply_change_fee(booking_reference)**.\n", "\n", "---\n", "\n", "By adhering to this updated policy and utilizing the specified functions appropriately, you will enhance efficiency and customer satisfaction while reducing errors in processing cancellations and changes.\n" ] }, { "data": { "text/markdown": [ "### Accuracy: 100.00%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>Could you please provide the new date you would like to change your flight to? Additionally, I need to check the type of ticket you have to proceed. I will do that now.</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>Since your ticket is non-refundable, you will be offered flight credit for future use. I'll proceed with that now.</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 True \n", "8 True \n", "9 True \n", "10 True \n", "11 True \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", "16 True \n", "\n", " assistant_message_content \n", "0 None \n", "1 Could you please provide the new date you would like to change your flight to? Additionally, I need to check the type of ticket you have to proceed. I will do that now. \n", "2 None \n", "3 Since your ticket is non-refundable, you will be offered flight credit for future use. I'll proceed with that now. \n", "4 None \n", "5 None \n", "6 None \n", "7 None \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: - **Under \"Minimal Dialogue\":**\n", " - **Do Not Mention Internal Processes:**\n", " - *Guideline:* Avoid stating internal actions or checks being performed. Focus communication on the customer's needs and the information required from them.\n", " - **Example Adjustment:**\n", " - Instead of saying, \"I need to check the type of ticket you have to proceed. I will do that now,\" simply proceed with the necessary function call or ask for essential information if required.\n", "\n", "- **Emphasize Clarity and Conciseness:**\n", " - Ensure that all communications are clear, necessary, and contribute directly to resolving the customer's request.\n" ] }, { "data": { "text/markdown": [ "### Accuracy: 82.35%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>Since you have a non-refundable ticket, please let me know how you would like to proceed. You may consider options such as offering a flight credit or assessing your eligibility for any potential exceptions.</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>process_partial_group_cancellation</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 process_change_no_fee \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 rebook_without_fee \n", "8 process_flexible_cancellation \n", "9 process_partial_group_cancellation \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 False \n", "2 True \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 False \n", "8 True \n", "9 False \n", "10 True \n", "11 True \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", "16 True \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 Since you have a non-refundable ticket, please let me know how you would like to proceed. You may consider options such as offering a flight credit or assessing your eligibility for any potential exceptions. \n", "4 None \n", "5 None \n", "6 None \n", "7 None \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: **Updated Internal Flight Cancellations and Changes Policy**\n", "\n", "**Purpose**: This document serves as a detailed guide for internal support agents to handle flight cancellations and changes. The focus is on providing clear instructions, ensuring efficiency, consistency, and customer satisfaction.\n", "\n", "## **Table of Contents**\n", "\n", "1. General Guidelines for Handling Customer Requests \n", "2. Functions and Procedures \n", "3. Cancellations: Types and Policies \n", "4. Changes: Types and Policies \n", "5. Rebooking Guidelines \n", "6. Compensation and Refund Rules \n", "7. Special Cases \n", "8. FAQs for Common Scenarios\n", "\n", "### **1\\. General Guidelines for Handling Customer Requests**\n", "\n", "- **Confirm Identity**: Verify the customer's identity by asking for their booking reference and any additional required details (e.g., name and flight number).\n", "\n", "- **Listen and Understand**: Determine if the customer wants to cancel, change, or inquire about compensation.\n", "\n", "- **Use Appropriate Functions**: Directly call the relevant function to process the customer's request, based on the policy guidelines.\n", "\n", "- **Follow Sequential Steps**: Ensure that prerequisite steps, such as checking the ticket type, are performed before proceeding with actions like processing changes or cancellations.\n", "\n", "- **Minimal Dialogue**: Avoid unnecessary messages or questions. Proceed with the required action unless additional information is needed.\n", "\n", "- **Be Empathetic**: Express empathy, especially when the customer is experiencing inconvenience due to the airline.\n", "\n", "### **2\\. Functions and Procedures**\n", "\n", "To ensure consistency and efficiency, use the following predefined functions when assisting customers:\n", "\n", "- **check_ticket_type(booking_reference)**: Determines the ticket type (non-refundable, refundable, or flexible).\n", "\n", "- **process_full_refund(booking_reference)**: Processes a full refund for the customer.\n", "\n", "- **offer_flight_credit(booking_reference)**: Offers flight credit for future use.\n", "\n", "- **apply_change_fee(booking_reference)**: Applies any applicable change fees to the booking.\n", "\n", "- **process_change_no_fee(booking_reference)**: Processes a change to the booking without any fees.\n", "\n", "- **rebook_without_fee(booking_reference)**: Rebooks the customer on the next available flight without additional charges.\n", "\n", "- **offer_accommodation(booking_reference)**: Arranges hotel accommodation for the customer.\n", "\n", "- **check_next_available_flight(booking_reference)**: Checks for the next available flight options.\n", "\n", "- **prioritize_missed_connections(booking_reference)**: Prioritizes rebooking for missed connections due to airline delays.\n", "\n", "- **check_compensation_eligibility(booking_reference)**: Checks if the customer is eligible for compensation.\n", "\n", "- **permit_name_change(booking_reference)**: Processes a name change on the booking.\n", "\n", "- **process_partial_group_cancellation(booking_reference)**: Processes individual cancellations within a group booking.\n", "\n", "- **process_flexible_cancellation(booking_reference, documentation)**: Processes cancellations with flexibility due to special circumstances (e.g., medical emergencies).\n", "\n", "**Function Calling Guidelines**:\n", "\n", "- **Direct Action**: When a policy dictates a specific action, call the corresponding function directly without unnecessary conversation.\n", "\n", "- **Sequence of Actions**: Ensure that functions are called in the correct order, especially when certain actions depend on the results of previous steps (e.g., **check_ticket_type** before processing changes).\n", "\n", "- **Information Gathering**: Only ask for additional information if it is essential to proceed.\n", "\n", "- **Efficiency**: Focus on resolving the customer's request promptly by using the appropriate function.\n", "\n", "### **3\\. Cancellations: Types and Policies**\n", "\n", "#### **3.1 Customer-Initiated Cancellations**\n", "\n", "- **Refundable Tickets**:\n", "\n", " - **Within 24 Hours of Booking**:\n", " - **Action**: Call **process_full_refund(booking_reference)**.\n", "\n", " - **Beyond 24 Hours**:\n", " - **Action**: First, call **check_ticket_type(booking_reference)** to confirm refundable status, then proceed with **process_full_refund(booking_reference)** if eligible.\n", "\n", "- **Non-Refundable Tickets**:\n", "\n", " - **Action**: Offer flight credit by calling **offer_flight_credit(booking_reference)**.\n", "\n", "- **Flexible Tickets**:\n", "\n", " - **Action**: Call **process_full_refund(booking_reference)** or **offer_flight_credit(booking_reference)** based on customer preference.\n", "\n", "### **4\\. Changes: Types and Policies**\n", "\n", "#### **4.1 Customer-Initiated Changes**\n", "\n", "- **Important**: Always **call **check_ticket_type(booking_reference)** before processing any changes** to determine the applicable policies based on the ticket type.\n", "\n", "- **Same-Day Changes**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: After confirming ticket type, call **process_change_no_fee(booking_reference)**.\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: After confirming ticket type, call **apply_change_fee(booking_reference)**.\n", "\n", "- **Changes in Advance**:\n", "\n", " - **Within 7 Days of Departure**:\n", " - **Action**: After confirming ticket type, call **apply_change_fee(booking_reference)**.\n", "\n", " - **Beyond 7 Days**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: Call **process_change_no_fee(booking_reference)**.\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **apply_change_fee(booking_reference)**.\n", "\n", "#### **4.2 Airline-Initiated Changes**\n", "\n", "- **Schedule Changes**:\n", "\n", " - **Minor Changes (< 2 Hours)**:\n", " - **Action**: Inform the customer of the new schedule. Only proceed if the customer requests alternatives.\n", "\n", " - **Major Changes (≥ 2 Hours)**:\n", " - **Action**: Offer options by calling:\n", " - **rebook_without_fee(booking_reference)**\n", " - **process_full_refund(booking_reference)**\n", " - **offer_flight_credit(booking_reference)**\n", "\n", "### **5\\. Rebooking Guidelines**\n", "\n", "- **Priority Handling**:\n", "\n", " - **Action**: Call **rebook_without_fee(booking_reference)** to rebook on the next available flight with our airline.\n", "\n", "- **Interline Agreements**:\n", "\n", " - **Action**: If no suitable options are available, check for flights with partner airlines. Proceed accordingly.\n", "\n", "- **Upgrade Availability**:\n", "\n", " - **Action**: If only higher-class seats are available, rebook without additional charge.\n", "\n", "### **6\\. Compensation and Refund Rules**\n", "\n", "#### **6.1 Refund Processing**\n", "\n", "- **Refund Timeline**:\n", "\n", " - **Action**: Inform the customer that refunds are processed within 7-10 business days for credit cards and 14-20 business days for debit cards.\n", "\n", "- **Refund Options**:\n", "\n", " - **Original Payment Method**:\n", "\n", " - **Action**: Default to refunding the original payment method.\n", "\n", " - **Travel Credits**:\n", "\n", " - **Action**: Call **offer_flight_credit(booking_reference)** if the customer prefers.\n", "\n", "#### **6.2 Compensation**\n", "\n", "- **Eligibility Check**:\n", "\n", " - **Action**: Call **check_compensation_eligibility(booking_reference)**.\n", "\n", "- **Meals and Accommodation**:\n", "\n", " - **Action**: If eligible, call **offer_accommodation(booking_reference)**.\n", "\n", "### **7\\. Special Cases**\n", "\n", "#### **7.1 Medical Emergencies**\n", "\n", "- **Flexible Cancellations**:\n", "\n", " - **Action**: Upon receiving a medical certificate, call **process_flexible_cancellation(booking_reference, documentation)**.\n", "\n", "#### **7.2 Bereavement Cases**\n", "\n", "- **Special Flexibility**:\n", "\n", " - **Action**: Call **process_flexible_cancellation(booking_reference, documentation)** when proper documentation is provided.\n", "\n", "#### **7.3 Group Bookings**\n", "\n", "- **Partial Cancellations**:\n", "\n", " - **Action**: Call **process_partial_group_cancellation(booking_reference)** for individual cancellations within a group booking.\n", "\n", "- **Name Changes**:\n", "\n", " - **Action**: If requested 7+ days before departure, call **permit_name_change(booking_reference)**.\n", "\n", "#### **7.4 Unaccompanied Minors**\n", "\n", "- **Rebooking with Supervision**:\n", "\n", " - **Action**: Call **rebook_without_fee(booking_reference)**, ensuring supervision arrangements are made.\n", "\n", "- **Priority for Minors**:\n", "\n", " - **Action**: Prioritize unaccompanied minors by calling **prioritize_missed_connections(booking_reference)** if necessary.\n", "\n", "### **8\\. FAQs for Common Scenarios**\n", "\n", "**Q1: What if a customer’s connecting flight is affected?**\n", "\n", "- **Action**: Call **prioritize_missed_connections(booking_reference)** to rebook the customer promptly.\n", "\n", "**Q2: How do I handle customers seeking upgrades after a cancellation?**\n", "\n", "- **Action**: If available, rebook using **rebook_without_fee(booking_reference)** and inform the customer of the upgrade at no extra cost.\n", "\n", "**Q3: What if a customer needs to change a destination?**\n", "\n", "- **Action**: Inform the customer about fare differences and any change fees. Proceed by calling **apply_change_fee(booking_reference)**.\n", "\n", "**Q4: What are the rules around rebooking for third-party bookings?**\n", "\n", "- **Action**: Advise the customer to contact the original booking agent. If eligible, proceed with rebooking per policy.\n", "\n", "**Q5: What should I do if the customer requests compensation for inconvenience?**\n", "\n", "- **Action**: Call **check_compensation_eligibility(booking_reference)** to assess and proceed accordingly.\n", "\n", "**Q6: How do I handle a situation where a customer’s preferred flight is fully booked?**\n", "\n", "- **Action**: Call **check_next_available_flight(booking_reference)** to offer alternative options. If possible, place the customer on a waitlist.\n", "\n", "**Q7: What if a customer wants to change the name on the ticket?**\n", "\n", "- **Action**: If the request is made 7+ days before departure, call **permit_name_change(booking_reference)**.\n", "\n", "**Q8: How should I proceed if the customer is unsure about accepting flight credit?**\n", "\n", "- **Action**: Offer to process a refund by calling **process_full_refund(booking_reference)** if policy allows.\n", "\n", "**Q9: What if the customer is a no-show due to unforeseen circumstances?**\n", "\n", "- **Action**: Assess the situation. If applicable, offer flight credit by calling **offer_flight_credit(booking_reference)**.\n", "\n", "**Q10: How do I handle requests for same-day flight changes?**\n", "\n", "- **Action**: Determine the ticket type by calling **check_ticket_type(booking_reference)**:\n", "\n", " - **Flexible Ticket**: Call **process_change_no_fee(booking_reference)**.\n", "\n", " - **Non-Flexible Ticket**: Call **apply_change_fee(booking_reference)**.\n", "\n", "By adhering to this updated policy, specifically ensuring that:\n", "\n", "- **Ticket Type Verification**: Always verify the ticket type using **check_ticket_type(booking_reference)** before processing changes or cancellations.\n", "\n", "- **Correct Function Usage**: Use the appropriate function for each scenario, distinguishing between similar actions (e.g., **process_change_no_fee** vs. **rebook_without_fee**).\n", "\n", "- **Sequential Steps**: Follow the correct sequence of actions required for each customer request.\n", "\n", "You will enhance efficiency and accuracy, reducing errors and improving performance against the dataset.\n" ] }, { "data": { "text/markdown": [ "### Accuracy: 100.00%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct assistant_message_content \n", "0 True None \n", "1 True None \n", "2 True None \n", "3 True None \n", "4 True None \n", "5 True None \n", "6 True None \n", "7 True None \n", "8 True None \n", "9 True None \n", "10 True None \n", "11 True None \n", "12 True None \n", "13 True None \n", "14 True None \n", "15 True None \n", "16 True None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: **Internal Flight Cancellations and Changes Policy**\n", "\n", "- **Sequential Function Calls:** Emphasized the importance of calling **`check_ticket_type(booking_reference)`** before processing any changes or cancellations. This ensures that agents handle requests according to the correct ticket policies.\n", "\n", "- **Correct Function Usage:** Clarified the distinction between similar functions (e.g., **`process_change_no_fee`** vs. **`rebook_without_fee`**) to prevent confusion and ensure appropriate actions are taken.\n", "\n", "- **Minimal Dialogue:** Reinforced guidance on avoiding unnecessary conversation and internal process mentions, focusing communication on essential information required from the customer.\n", "\n", "- **Comprehensive Function List:** Included all relevant functions, such as **`process_partial_group_cancellation(booking_reference)`**, to cover various scenarios agents may encounter.\n", "\n", "- **Action-Oriented Instructions:** Provided clear, direct actions for agents to follow in each policy section, facilitating efficient and accurate responses.\n" ] }, { "data": { "text/markdown": [ "### Accuracy: 94.12%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>Your ticket is non-refundable. Unfortunately, this means that I cannot process a cancellation for you. Would you like to explore alternate options, such as flight credit or changes with applicable fees?</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>False</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 rebook_without_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 False \n", "8 True \n", "9 True \n", "10 True \n", "11 True \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", "16 True \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 Your ticket is non-refundable. Unfortunately, this means that I cannot process a cancellation for you. Would you like to explore alternate options, such as flight credit or changes with applicable fees? \n", "4 None \n", "5 None \n", "6 None \n", "7 None \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: ## **Policies for Flight Cancellations and Changes**\n", "\n", "### **3\\. Cancellations: Types and Policies**\n", "\n", "#### **3.1 Customer-Initiated Cancellations**\n", "\n", "- **Refundable Tickets**:\n", "\n", " - **Within 24 Hours of Booking**:\n", " - **Action**: Call **`process_full_refund(booking_reference)`**.\n", "\n", " - **Beyond 24 Hours**:\n", " - **Action**: First, call **`check_ticket_type(booking_reference)`** to confirm refundable status, then proceed with **`process_full_refund(booking_reference)`** if eligible.\n", "\n", "- **Non-Refundable Tickets**:\n", "\n", " - **Action**: Offer flight credit by calling **`offer_flight_credit(booking_reference)`**.\n", "\n", "- **Flexible Tickets**:\n", "\n", " - **Action**: Call **`process_full_refund(booking_reference)`** or **`offer_flight_credit(booking_reference)`** based on customer preference.\n", "\n", "### **4\\. Changes: Types and Policies**\n", "\n", "#### **4.1 Customer-Initiated Changes**\n", "\n", "- **Important**: Always **call **`check_ticket_type(booking_reference)`** before processing any changes** to determine the applicable policies based on the ticket type.\n", "\n", "- **Same-Day Changes**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: After confirming ticket type, call **`process_change_no_fee(booking_reference)`** to process the change without any fees.\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: After confirming ticket type, call **`apply_change_fee(booking_reference)`** to process the change with applicable fees.\n", "\n", "- **Changes in Advance**:\n", "\n", " - **Within 7 Days of Departure**:\n", " - **Action**: After confirming ticket type, call **`apply_change_fee(booking_reference)`**.\n", "\n", " - **Beyond 7 Days**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", "\n", "#### **4.2 Airline-Initiated Changes**\n", "\n", "- **Schedule Changes**:\n", "\n", " - **Minor Changes (< 2 Hours)**:\n", " - **Action**: Inform the customer of the new schedule. Only proceed if the customer requests alternatives.\n", "\n", " - **Major Changes (≥ 2 Hours)**:\n", " - **Action**: Offer options by calling:\n", "\n", " - **`rebook_without_fee(booking_reference)`**\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", "\n", "- **Airline-Initiated Disruptions**:\n", "\n", " - **Action**: For cancellations or significant delays initiated by the airline, rebook the customer without additional charges by calling **`rebook_without_fee(booking_reference)`**.\n", "\n", "### **5\\. Rebooking Guidelines**\n", "\n", "- **Priority Handling**:\n", "\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`** to rebook on the next available flight with our airline, especially in cases of airline-initiated disruptions.\n", "\n", "- **Interline Agreements**:\n", "\n", " - **Action**: If no suitable options are available, check for flights with partner airlines. Proceed accordingly.\n", "\n", "- **Upgrade Availability**:\n", "\n", " - **Action**: If only higher-class seats are available, rebook without additional charge.\n", "\n", "### **6\\. Compensation and Refund Rules**\n", "\n", "#### **6.1 Refund Processing**\n", "\n", "- **Refund Timeline**:\n", "\n", " - **Action**: Inform the customer that refunds are processed within 7-10 business days for credit cards and 14-20 business days for debit cards.\n", "\n", "- **Refund Options**:\n", "\n", " - **Original Payment Method**:\n", "\n", " - **Action**: Default to refunding the original payment method.\n", "\n", " - **Travel Credits**:\n", "\n", " - **Action**: Call **`offer_flight_credit(booking_reference)`** if the customer prefers.\n", "\n", "#### **6.2 Compensation**\n", "\n", "- **Eligibility Check**:\n", "\n", " - **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", "\n", "- **Meals and Accommodation**:\n", "\n", " - **Action**: If eligible, call **`offer_accommodation(booking_reference)`**.\n", "\n", "### **7\\. Special Cases**\n", "\n", "#### **7.1 Medical Emergencies**\n", "\n", "- **Flexible Cancellations**:\n", "\n", " - **Action**: Upon receiving a medical certificate, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "\n", "#### **7.2 Bereavement Cases**\n", "\n", "- **Special Flexibility**:\n", "\n", " - **Action**: Call **`process_flexible_cancellation(booking_reference, documentation)`** when proper documentation is provided.\n", "\n", "#### **7.3 Group Bookings**\n", "\n", "- **Partial Cancellations**:\n", "\n", " - **Action**: Call **`process_partial_group_cancellation(booking_reference)`** for individual cancellations within a group booking.\n", "\n", "- **Name Changes**:\n", "\n", " - **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "\n", "#### **7.4 Unaccompanied Minors**\n", "\n", "- **Rebooking with Supervision**:\n", "\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`**, ensuring supervision arrangements are made.\n", "\n", "- **Priority for Minors**:\n", "\n", " - **Action**: Prioritize unaccompanied minors by calling **`prioritize_missed_connections(booking_reference)`** if necessary.\n", "\n", "### **8\\. FAQs for Common Scenarios**\n", "\n", "**Q1: What if a customer’s connecting flight is affected?**\n", "\n", "- **Action**: Call **`prioritize_missed_connections(booking_reference)`** to rebook the customer promptly.\n", "\n", "---\n", "\n", "**Q2: How do I handle customers seeking upgrades after a cancellation?**\n", "\n", "- **Action**: If available, rebook using **`rebook_without_fee(booking_reference)`** and inform the customer of the upgrade at no extra cost.\n", "\n", "---\n", "\n", "**Q3: What if a customer needs to change a destination?**\n", "\n", "- **Action**: Inform the customer about fare differences and any change fees. Proceed by calling **`apply_change_fee(booking_reference)`**.\n", "\n", "---\n", "\n", "**Q4: What are the rules around rebooking for third-party bookings?**\n", "\n", "- **Action**: Advise the customer to contact the original booking agent. If eligible, proceed with rebooking per policy.\n", "\n", "---\n", "\n", "**Q5: What should I do if the customer requests compensation for inconvenience?**\n", "\n", "- **Action**: Call **`check_compensation_eligibility(booking_reference)`** to assess and proceed accordingly.\n", "\n", "---\n", "\n", "**Q6: How do I handle a situation where a customer’s preferred flight is fully booked?**\n", "\n", "- **Action**: Call **`check_next_available_flight(booking_reference)`** to offer alternative options. If possible, place the customer on a waitlist.\n", "\n", "---\n", "\n", "**Q7: What if a customer wants to change the name on the ticket?**\n", "\n", "- **Action**: If the request is made 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "\n", "---\n", "\n", "**Q8: How should I proceed if the customer is unsure about accepting flight credit?**\n", "\n", "- **Action**: Offer to process a refund by calling **`process_full_refund(booking_reference)`** if policy allows.\n", "\n", "---\n", "\n", "**Q9: What if the customer is a no-show due to unforeseen circumstances?**\n", "\n", "- **Action**: Assess the situation. If applicable, offer flight credit by calling **`offer_flight_credit(booking_reference)`**.\n", "\n", "---\n", "\n", "**Q10: How do I handle requests for same-day flight changes?**\n", "\n", "- **Action**: Determine the ticket type by calling **`check_ticket_type(booking_reference)`**:\n", "\n", " - **Flexible Ticket**: Call **`process_change_no_fee(booking_reference)`**.\n", "\n", " - **Non-Flexible Ticket**: Call **`apply_change_fee(booking_reference)`**.\n" ] } ], "source": [ "updated_policy = flight_cancellation_routine\n", "messages = [\n", " {\n", " \"role\": \"user\",\n", " \"content\": f\"\"\"\n", "You are an agent that is responsible for improving the quality of routine instructions that are provided to a customer service LLM agent.\n", "\n", "I am going to give you the policy for the customer service agent that contains detailed instructions on how to handle flight cancellations and changes.\n", "\n", "You will also be provided with the results from an eval set that include the following:\n", " - conversation history: This is the conversation that we present to the LLM along with the system prompt\n", " - expected_function: This is the function we expect the LLM to call\n", " - expected_input: This is the input we expect the LLM to provide to the function\n", " - actual_function: This is the actual function the LLM called\n", " - actual_input: This is the actual input the LLM provided\n", " - assistant_message_content: This is the message the LLM generated when it returned its response\n", " - is_correct: True/False value depending on if the model responded correctly\n", "\n", "Carefully analyze the instructions provided as well as the results of the eval. Get a firm understanding of the failures in the policy.\n", "\n", "Return an updated policy that will perform better against the dataset.\n", "\n", "Here is the current policy:\n", "{flight_cancellation_policy}\n", "\"\"\"\n", " }\n", "]\n", "\n", "for _ in range(5):\n", " # Evaluate the function calls with the current policy\n", " df, accuracy = evaluate_function_calls('evals/functionCallingEval.csv', updated_policy, 'gpt-4o-mini-2024-07-18')\n", " \n", " # Display the accuracy as a mini header\n", " display(Markdown(f\"### Accuracy: {accuracy:.2%}\"))\n", " display(df)\n", " results_json = df.to_json(orient='records')\n", "\n", " messages.append({\n", " \"role\": \"user\",\n", " \"content\": f\"\"\"\n", "Here are the results based on the current policy:\n", "{results_json}\n", "\"\"\"\n", " })\n", " # Use the metaprompt function to get an updated policy\n", " temp_policy_json = enforce_schema(metaprompt(messages))\n", " temp_policy_str = temp_policy_json.strip(\"json```\").strip(\"```\")\n", " temp_policy = json.loads(temp_policy_str)[\"final_answer\"]\n", " print(f\"Corrected Policy: {temp_policy}\")\n", "\n", " messages.append({\n", " \"role\": \"assistant\",\n", " \"content\": f\"\"\"\n", "{temp_policy}\n", "\"\"\"\n", " })\n", "\n", " # Update the policy for the next iteration\n", " updated_policy = temp_policy\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Distilling Down to a smaller model" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each time we release a new snapshot of a model, it is always a challenge to ensure that your existing prompt works for the new snapshot.\n", "\n", "In this example, we'll simulate that work by trying to get the routine to work for our older GPT 3.5 Turbo model." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "### Accuracy: 94.12%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>I have confirmed that you have a non-refundable ticket. Since your ticket is non-refundable, I can offer you flight credit for future use. Would you like to proceed with flight credit?</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>False</td>\n", " <td>Great news! Your ticket is a flexible ticket, which means we can proceed with rebooking your flight without any additional fees. Let's go ahead and rebook your flight on the next available option.</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 rebook_without_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 False \n", "8 True \n", "9 True \n", "10 True \n", "11 True \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", "16 True \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 I have confirmed that you have a non-refundable ticket. Since your ticket is non-refundable, I can offer you flight credit for future use. Would you like to proceed with flight credit? \n", "4 None \n", "5 None \n", "6 None \n", "7 Great news! Your ticket is a flexible ticket, which means we can proceed with rebooking your flight without any additional fees. Let's go ahead and rebook your flight on the next available option. \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: ## **Policies for Flight Cancellations and Changes**\n", "\n", "### **2\\. Important Clarifications**\n", "\n", "#### **2.1 Difference Between `process_change_no_fee` and `rebook_without_fee`**\n", "\n", "- **`process_change_no_fee(booking_reference)`**:\n", " - **Use When**: The customer initiates a change to their booking and holds a **flexible ticket**.\n", " - **Purpose**: Processes the customer's requested change **without any additional fees**.\n", " - **Examples**:\n", " - Customer wants to change the departure time or date.\n", " - Customer wishes to change to a different flight on the same day.\n", "\n", "- **`rebook_without_fee(booking_reference)`**:\n", " - **Use When**: The **airline initiates a change** or there is an **airline-induced disruption** (e.g., cancellations, significant delays).\n", " - **Purpose**: Rebooks the customer on an alternative flight **without any additional charges**.\n", " - **Examples**:\n", " - Flight is canceled by the airline.\n", " - Flight is significantly delayed (≥ 2 hours).\n", " - Offering an upgrade after a cancellation (see **FAQ Q2**).\n", "\n", "**Remember**: It's crucial to distinguish between customer-initiated and airline-initiated changes to select the appropriate function.\n", "\n", "---\n", "\n", "### **3\\. Cancellations: Types and Policies**\n", "\n", "#### **3.1 Customer-Initiated Cancellations**\n", "\n", "- **Refundable Tickets**:\n", "\n", " - **Within 24 Hours of Booking**:\n", " - **Action**: Call **`process_full_refund(booking_reference)`**.\n", "\n", " - **Beyond 24 Hours**:\n", " - **Action**: First, call **`check_ticket_type(booking_reference)`** to confirm refundable status, then proceed with **`process_full_refund(booking_reference)`** if eligible.\n", "\n", "- **Non-Refundable Tickets**:\n", "\n", " - **Action**: Offer flight credit by calling **`offer_flight_credit(booking_reference)`**.\n", "\n", "- **Flexible Tickets**:\n", "\n", " - **Action**: Call **`process_full_refund(booking_reference)`** or **`offer_flight_credit(booking_reference)`** based on customer preference.\n", "\n", "### **4\\. Changes: Types and Policies**\n", "\n", "#### **4.1 Customer-Initiated Changes**\n", "\n", "- **Important**: Always **call **`check_ticket_type(booking_reference)`** before processing any changes** to determine the applicable policies based on the ticket type.\n", "\n", "- **Same-Day Changes**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: After confirming ticket type, call **`process_change_no_fee(booking_reference)`** to process the change without any fees.\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: After confirming ticket type, call **`apply_change_fee(booking_reference)`** to process the change with applicable fees.\n", "\n", "- **Changes in Advance**:\n", "\n", " - **Within 7 Days of Departure**:\n", " - **Action**: After confirming ticket type, call **`apply_change_fee(booking_reference)`**.\n", "\n", " - **Beyond 7 Days**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", "\n", "#### **4.2 Airline-Initiated Changes**\n", "\n", "- **Schedule Changes**:\n", "\n", " - **Minor Changes (< 2 Hours)**:\n", " - **Action**: Inform the customer of the new schedule. Only proceed if the customer requests alternatives.\n", "\n", " - **Major Changes (≥ 2 Hours)**:\n", " - **Action**: Offer options by calling:\n", "\n", " - **`rebook_without_fee(booking_reference)`**\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", "\n", "- **Airline-Initiated Disruptions**:\n", "\n", " - **Action**: For cancellations or significant delays initiated by the airline, rebook the customer without additional charges by calling **`rebook_without_fee(booking_reference)`**.\n", "\n", "### **5\\. Rebooking Guidelines**\n", "\n", "- **Priority Handling**:\n", "\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`** to rebook on the next available flight with our airline, especially in cases of airline-initiated disruptions.\n", "\n", "- **Interline Agreements**:\n", "\n", " - **Action**: If no suitable options are available, check for flights with partner airlines. Proceed accordingly.\n", "\n", "- **Upgrade Availability**:\n", "\n", " - **Action**: If only higher-class seats are available, rebook without additional charge.\n", "\n", "### **6\\. Compensation and Refund Rules**\n", "\n", "#### **6.1 Refund Processing**\n", "\n", "- **Refund Timeline**:\n", "\n", " - **Action**: Inform the customer that refunds are processed within 7-10 business days for credit cards and 14-20 business days for debit cards.\n", "\n", "- **Refund Options**:\n", "\n", " - **Original Payment Method**:\n", "\n", " - **Action**: Default to refunding the original payment method.\n", "\n", " - **Travel Credits**:\n", "\n", " - **Action**: Call **`offer_flight_credit(booking_reference)`** if the customer prefers.\n", "\n", "#### **6.2 Compensation**\n", "\n", "- **Eligibility Check**:\n", "\n", " - **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", "\n", "- **Meals and Accommodation**:\n", "\n", " - **Action**: If eligible, call **`offer_accommodation(booking_reference)`**.\n", "\n", "### **7\\. Special Cases**\n", "\n", "#### **7.1 Medical Emergencies**\n", "\n", "- **Flexible Cancellations**:\n", "\n", " - **Action**: Upon receiving a medical certificate, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "\n", "#### **7.2 Bereavement Cases**\n", "\n", "- **Special Flexibility**:\n", "\n", " - **Action**: Call **`process_flexible_cancellation(booking_reference, documentation)`** when proper documentation is provided.\n", "\n", "#### **7.3 Group Bookings**\n", "\n", "- **Partial Cancellations**:\n", "\n", " - **Action**: Call **`process_partial_group_cancellation(booking_reference)`** for individual cancellations within a group booking.\n", "\n", "- **Name Changes**:\n", "\n", " - **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "\n", "#### **7.4 Unaccompanied Minors**\n", "\n", "- **Rebooking with Supervision**:\n", "\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`**, ensuring supervision arrangements are made.\n", "\n", "- **Priority for Minors**:\n", "\n", " - **Action**: Prioritize unaccompanied minors by calling **`prioritize_missed_connections(booking_reference)`** if necessary.\n", "\n", "### **8\\. FAQs for Common Scenarios**\n", "\n", "**Q1: What if a customer’s connecting flight is affected?**\n", "\n", "- **Action**: Call **`prioritize_missed_connections(booking_reference)`** to rebook the customer promptly.\n", "\n", "---\n", "\n", "**Q2: How do I handle customers seeking upgrades after a cancellation?**\n", "\n", "- **Action**: If available, rebook using **`rebook_without_fee(booking_reference)`** and inform the customer of the upgrade at no extra cost.\n", "\n", "---\n", "\n", "**Q3: What if a customer needs to change a destination?**\n", "\n", "- **Action**: Inform the customer about fare differences and any change fees. Proceed by calling **`apply_change_fee(booking_reference)`**.\n", "\n", "---\n", "\n", "**Q4: What are the rules around rebooking for third-party bookings?**\n", "\n", "- **Action**: Advise the customer to contact the original booking agent. If eligible, proceed with rebooking per policy.\n", "\n", "---\n", "\n", "**Q5: What should I do if the customer requests compensation for inconvenience?**\n", "\n", "- **Action**: Call **`check_compensation_eligibility(booking_reference)`** to assess and proceed accordingly.\n", "\n", "---\n", "\n", "**Q6: How do I handle a situation where a customer’s preferred flight is fully booked?**\n", "\n", "- **Action**: Call **`check_next_available_flight(booking_reference)`** to offer alternative options. If possible, place the customer on a waitlist.\n", "\n", "---\n", "\n", "**Q7: What if a customer wants to change the name on the ticket?**\n", "\n", "- **Action**: If the request is made 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "\n", "---\n", "\n", "**Q8: How should I proceed if the customer is unsure about accepting flight credit?**\n", "\n", "- **Action**: Offer to process a refund by calling **`process_full_refund(booking_reference)`** if policy allows.\n", "\n", "---\n", "\n", "**Q9: What if the customer is a no-show due to unforeseen circumstances?**\n", "\n", "- **Action**: Assess the situation. If applicable, offer flight credit by calling **`offer_flight_credit(booking_reference)`**.\n", "\n", "---\n", "\n", "**Q10: How do I handle requests for same-day flight changes?**\n", "\n", "- **Action**: Determine the ticket type by calling **`check_ticket_type(booking_reference)`**:\n", "\n", " - **Flexible Ticket**: Call **`process_change_no_fee(booking_reference)`**.\n", "\n", " - **Non-Flexible Ticket**: Call **`apply_change_fee(booking_reference)`**.\n", "\n", "---\n", "\n", "**Note**: Always ensure that the correct function is used based on whether the change is **customer-initiated** or **airline-initiated** to provide accurate assistance and comply with company policies.\n" ] }, { "data": { "text/markdown": [ "### Accuracy: 94.12%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>Your ticket is non-refundable. In this case, I can offer you flight credit for future use. Would you like to proceed with that option?</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>False</td>\n", " <td>Great news! You have a flexible ticket, which means you are eligible for rebooking without any additional fees. Let's proceed with rebooking your flight.</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 rebook_without_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 False \n", "8 True \n", "9 True \n", "10 True \n", "11 True \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", "16 True \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 Your ticket is non-refundable. In this case, I can offer you flight credit for future use. Would you like to proceed with that option? \n", "4 None \n", "5 None \n", "6 None \n", "7 Great news! You have a flexible ticket, which means you are eligible for rebooking without any additional fees. Let's proceed with rebooking your flight. \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: ## **Policies for Flight Cancellations and Changes**\n", "\n", "### **2. Important Clarifications**\n", "\n", "#### **2.1 Distinguishing Between Customer-Initiated and Airline-Initiated Actions**\n", "\n", "- **Customer-Initiated Actions**:\n", " - **Definition**: Actions where the customer requests changes or cancellations to their booking.\n", " - **Functions to Use**:\n", " - **`process_change_no_fee(booking_reference)`**\n", " - **`apply_change_fee(booking_reference)`**\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", "\n", "- **Airline-Initiated Actions**:\n", " - **Definition**: Actions where the airline changes or cancels flights due to operational reasons.\n", " - **Functions to Use**:\n", " - **`rebook_without_fee(booking_reference)`**\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", "\n", "#### **2.2 Clarifying Function Usage: `process_change_no_fee` vs. `rebook_without_fee`**\n", "\n", "- **`process_change_no_fee(booking_reference)`**:\n", " - **Use When**:\n", " - The **customer initiates** a change (e.g., flight time, date).\n", " - The customer holds a **flexible ticket**.\n", " - **Purpose**: To process the customer's requested change **without any additional fees**.\n", " - **Do Not Use**: For airline-initiated disruptions or rebookings.\n", "\n", "- **`rebook_without_fee(booking_reference)`**:\n", " - **Use When**:\n", " - The **airline initiates** a change due to cancellations, significant delays, or disruptions.\n", " - **Purpose**: To rebook the customer on an alternative flight **without any charges**.\n", " - **Do Not Use**: For customer-initiated changes, regardless of ticket type.\n", "\n", "**Important Note**: Even if the customer is not charged any fees, always use **`process_change_no_fee`** for **customer-initiated** changes.\n", "\n", "#### **2.3 Language and Terminology**\n", "\n", "- **Use of Terms**:\n", " - **\"Change\"**:\n", " - **Context**: Customer-initiated modifications to their booking.\n", " - **Examples**:\n", " - \"I would like to **change** my flight date.\"\n", " - \"Can I **change** my departure time?\"\n", " - **Associated Functions**:\n", " - **`process_change_no_fee(booking_reference)`**\n", " - **`apply_change_fee(booking_reference)`**\n", " - **\"Rebook\"**:\n", " - **Context**: Airline-initiated adjustments due to disruptions.\n", " - **Examples**:\n", " - \"We need to **rebook** you due to a flight cancellation.\"\n", " - \"We will **rebook** you on the next available flight.\"\n", " - **Associated Functions**:\n", " - **`rebook_without_fee(booking_reference)`**\n", "\n", "- **Avoiding Confusion**:\n", " - Do not use \"rebook\" when the customer is requesting a change.\n", " - Do not use \"change\" when the airline is adjusting the booking.\n", "\n", "#### **2.4 Examples of Correct Function Usage**\n", "\n", "- **Customer-Initiated Change (Flexible Ticket)**:\n", " - **Scenario**: A customer wants to change their flight to an earlier time.\n", " - **Action Steps**:\n", " 1. Call **`check_ticket_type(booking_reference)`** to confirm ticket flexibility.\n", " 2. Confirm the ticket is **flexible**.\n", " 3. Use **`process_change_no_fee(booking_reference)`** to process the change.\n", " - **Assistant Language**:\n", " - \"Since you have a flexible ticket, I can help you **change** your flight without any additional fees.\"\n", "\n", "- **Airline-Initiated Rebooking (Flight Cancellation)**:\n", " - **Scenario**: The airline has canceled the customer's flight.\n", " - **Action Steps**:\n", " 1. Inform the customer about the cancellation.\n", " 2. Use **`rebook_without_fee(booking_reference)`** to rebook them.\n", " - **Assistant Language**:\n", " - \"We apologize, but your flight has been canceled. I will **rebook** you on the next available flight at no additional charge.\"\n", "\n", "#### **2.5 Quick Reference Table**\n", "\n", "| **Scenario** | **Initiated By** | **Function to Use** |\n", "|-------------------------------------------|------------------|--------------------------------------|\n", "| Customer wants to change flight (no fee) | Customer | `process_change_no_fee` |\n", "| Customer wants to change flight (fee) | Customer | `apply_change_fee` |\n", "| Airline cancels flight | Airline | `rebook_without_fee` |\n", "| Airline delays flight significantly | Airline | `rebook_without_fee` |\n", "| Customer requests cancellation (eligible) | Customer | `process_full_refund` or `offer_flight_credit` |\n", "| Airline changes schedule (minor) | Airline | Inform customer; proceed if requested |\n", "\n", "### **3. Cancellations: Types and Policies**\n", "\n", "#### **3.1 Customer-Initiated Cancellations**\n", "\n", "- **Refundable Tickets**:\n", "\n", " - **Within 24 Hours of Booking**:\n", " - **Action**: Call **`process_full_refund(booking_reference)`**.\n", "\n", " - **Beyond 24 Hours**:\n", " - **Action**:\n", " 1. Call **`check_ticket_type(booking_reference)`** to confirm refundable status.\n", " 2. Proceed with **`process_full_refund(booking_reference)`** if eligible.\n", "\n", "- **Non-Refundable Tickets**:\n", "\n", " - **Action**: Offer flight credit by calling **`offer_flight_credit(booking_reference)`**.\n", "\n", "- **Flexible Tickets**:\n", "\n", " - **Action**: Offer options based on customer preference:\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", "\n", "### **4. Changes: Types and Policies**\n", "\n", "#### **4.1 Customer-Initiated Changes**\n", "\n", "- **Important**: Always **call `check_ticket_type(booking_reference)` before processing any changes** to determine the applicable policies.\n", "\n", "- **Same-Day Changes**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**:\n", " - After confirming ticket type, call **`process_change_no_fee(booking_reference)`**.\n", " - **Use Language**: \"I can assist you with changing your flight without any additional fees.\"\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**:\n", " - After confirming ticket type, call **`apply_change_fee(booking_reference)`**.\n", " - **Use Language**: \"I can help you change your flight. Please note that a change fee will apply.\"\n", "\n", "- **Changes in Advance**:\n", "\n", " - **Within 7 Days of Departure**:\n", " - **Action**: After confirming ticket type, call **`apply_change_fee(booking_reference)`**.\n", "\n", " - **Beyond 7 Days**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", "\n", "#### **4.2 Airline-Initiated Changes**\n", "\n", "- **Schedule Changes**:\n", "\n", " - **Minor Changes (< 2 Hours)**:\n", " - **Action**: Inform the customer of the new schedule. Proceed if the customer requests alternatives.\n", "\n", " - **Major Changes (≥ 2 Hours)**:\n", " - **Action**: Offer options by calling:\n", " - **`rebook_without_fee(booking_reference)`**\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", "\n", "- **Airline-Initiated Disruptions**:\n", "\n", " - **Action**: For cancellations or significant delays, rebook the customer by calling **`rebook_without_fee(booking_reference)`**.\n", "\n", "### **5. Rebooking Guidelines**\n", "\n", "- **Priority Handling**:\n", "\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`** for airline-initiated disruptions.\n", "\n", "- **Interline Agreements**:\n", "\n", " - **Action**: If no suitable options are available with our airline, check partner airlines.\n", "\n", "- **Upgrade Availability**:\n", "\n", " - **Action**: If only higher-class seats are available, rebook without additional charge using **`rebook_without_fee(booking_reference)`**.\n", "\n", "### **6. Compensation and Refund Rules**\n", "\n", "#### **6.1 Refund Processing**\n", "\n", "- **Refund Timeline**:\n", "\n", " - **Action**: Inform the customer about the refund processing time.\n", "\n", "- **Refund Options**:\n", "\n", " - **Original Payment Method**:\n", "\n", " - **Action**: Default to refunding the original payment method.\n", "\n", " - **Travel Credits**:\n", "\n", " - **Action**: Call **`offer_flight_credit(booking_reference)`** if the customer prefers.\n", "\n", "#### **6.2 Compensation**\n", "\n", "- **Eligibility Check**:\n", "\n", " - **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", "\n", "- **Meals and Accommodation**:\n", "\n", " - **Action**: If eligible, call **`offer_accommodation(booking_reference)`**.\n", "\n", "### **7. Special Cases**\n", "\n", "#### **7.1 Medical Emergencies**\n", "\n", "- **Flexible Cancellations**:\n", "\n", " - **Action**: Upon receiving documentation, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "\n", "#### **7.2 Bereavement Cases**\n", "\n", "- **Special Flexibility**:\n", "\n", " - **Action**: Call **`process_flexible_cancellation(booking_reference, documentation)`** with proper documentation.\n", "\n", "#### **7.3 Group Bookings**\n", "\n", "- **Partial Cancellations**:\n", "\n", " - **Action**: Call **`process_partial_group_cancellation(booking_reference)`**.\n", "\n", "- **Name Changes**:\n", "\n", " - **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "\n", "#### **7.4 Unaccompanied Minors**\n", "\n", "- **Rebooking with Supervision**:\n", "\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`**, ensuring supervision is arranged.\n", "\n", "- **Priority for Minors**:\n", "\n", " - **Action**: Prioritize by calling **`prioritize_missed_connections(booking_reference)`**.\n", "\n", "### **8. FAQs for Common Scenarios**\n", "\n", "**Q1: What if a customer’s connecting flight is affected?**\n", "\n", "- **Action**: Call **`prioritize_missed_connections(booking_reference)`**.\n", "\n", "**Q2: How do I handle customers seeking upgrades after a cancellation?**\n", "\n", "- **Action**: Use **`rebook_without_fee(booking_reference)`** for airline-initiated cancellations.\n", "\n", "**Q3: What if a customer needs to change a destination?**\n", "\n", "- **Action**: Inform about fare differences and fees. Proceed by calling **`apply_change_fee(booking_reference)`**.\n", "\n", "**Q4: What are the rules around rebooking for third-party bookings?**\n", "\n", "- **Action**: Advise contacting the original booking agent. Proceed per policy if eligible.\n", "\n", "**Q5: What should I do if the customer requests compensation for inconvenience?**\n", "\n", "- **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", "\n", "**Q6: How do I handle a situation where a customer’s preferred flight is fully booked?**\n", "\n", "- **Action**: Call **`check_next_available_flight(booking_reference)`**.\n", "\n", "**Q7: What if a customer wants to change the name on the ticket?**\n", "\n", "- **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "\n", "**Q8: How should I proceed if the customer is unsure about accepting flight credit?**\n", "\n", "- **Action**: Offer a refund by calling **`process_full_refund(booking_reference)`** if policy allows.\n", "\n", "**Q9: What if the customer is a no-show due to unforeseen circumstances?**\n", "\n", "- **Action**: Assess the situation. If applicable, call **`offer_flight_credit(booking_reference)`**.\n", "\n", "**Q10: How do I handle requests for same-day flight changes?**\n", "\n", "- **Action**:\n", "\n", " - Call **`check_ticket_type(booking_reference)`**.\n", " - For **flexible tickets**, call **`process_change_no_fee(booking_reference)`**.\n", " - For **non-flexible tickets**, call **`apply_change_fee(booking_reference)`**.\n", "\n", "**Q11: The customer has a flexible ticket and wants to change their flight without any fees. Which function should I call?**\n", "\n", "- **Action**: Call **`process_change_no_fee(booking_reference)`**. **Do not use** **`rebook_without_fee`** for customer-initiated changes.\n", "\n", "**Note**: Always ensure that the correct function is used based on whether the action is **customer-initiated** or **airline-initiated** to provide accurate assistance and comply with company policies.\n" ] }, { "data": { "text/markdown": [ "### Accuracy: 94.12%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>I have confirmed that you have a non-refundable ticket. Since the ticket is non-refundable, I will proceed with offering you flight credit for future use. Let me process that for you.</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>False</td>\n", " <td>Great news! You have a flexible ticket, which means you can rebook your flight without any additional fees. Let's proceed with the rebooking process.</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 rebook_without_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 False \n", "8 True \n", "9 True \n", "10 True \n", "11 True \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", "16 True \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 I have confirmed that you have a non-refundable ticket. Since the ticket is non-refundable, I will proceed with offering you flight credit for future use. Let me process that for you. \n", "4 None \n", "5 None \n", "6 None \n", "7 Great news! You have a flexible ticket, which means you can rebook your flight without any additional fees. Let's proceed with the rebooking process. \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: ## **Policies for Flight Cancellations and Changes**\n", "\n", "### **3. Cancellations: Types and Policies**\n", "\n", "#### **3.1 Customer-Initiated Cancellations**\n", "\n", "- **Refundable Tickets**:\n", "\n", " - **Within 24 Hours of Booking**:\n", " - **Action**: Call **`process_full_refund(booking_reference)`**.\n", "\n", " - **Beyond 24 Hours**:\n", " - **Action**:\n", " 1. Call **`check_ticket_type(booking_reference)`** to confirm refundable status.\n", " 2. Proceed with **`process_full_refund(booking_reference)`** if eligible.\n", "\n", "- **Non-Refundable Tickets**:\n", "\n", " - **Action**: Offer flight credit by calling **`offer_flight_credit(booking_reference)`**.\n", "\n", "- **Flexible Tickets**:\n", "\n", " - **Action**: Offer options based on customer preference:\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", "\n", "### **4. Changes: Types and Policies**\n", "\n", "#### **4.1 Customer-Initiated Changes**\n", "\n", "- **Important**: Always **call `check_ticket_type(booking_reference)` before processing any changes**.\n", "\n", "- **Same-Day Changes**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**:\n", " - After confirming ticket type, call **`process_change_no_fee(booking_reference)`**.\n", " - **Language**:\n", " - Use \"change\" in communication.\n", " - **Example**: \"You can **change** your flight without any additional fees.\"\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**:\n", " - After confirming ticket type, call **`apply_change_fee(booking_reference)`**.\n", " - **Language**:\n", " - Use \"change\" in communication.\n", " - **Example**: \"You can **change** your flight. Please note that a change fee will apply.\"\n", "\n", "- **Changes in Advance**:\n", "\n", " - **Within 7 Days of Departure**:\n", " - **Action**: After confirming ticket type, call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: Use \"change\".\n", "\n", " - **Beyond 7 Days**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", " - **Language**: Use \"change\".\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: Use \"change\".\n", "\n", "#### **4.2 Airline-Initiated Changes**\n", "\n", "- **Schedule Changes**:\n", "\n", " - **Minor Changes (< 2 Hours)**:\n", " - **Action**: Inform the customer of the new schedule.\n", " - **Proceed if Requested**: If the customer requests alternatives, assist accordingly.\n", "\n", " - **Major Changes (≥ 2 Hours)**:\n", " - **Action**: Offer options by calling:\n", " - **`rebook_without_fee(booking_reference)`**\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", " - **Language**:\n", " - Use \"rebook\" when offering alternative flights.\n", "\n", "- **Airline-Initiated Disruptions**:\n", "\n", " - **Action**: Rebook the customer by calling **`rebook_without_fee(booking_reference)`**.\n", " - **Language**: Use \"rebook\".\n", "\n", "### **5. Rebooking Guidelines**\n", "\n", "- **Priority Handling**:\n", "\n", " - **Action**: Use **`rebook_without_fee(booking_reference)`** for airline-initiated disruptions.\n", "\n", "- **Interline Agreements**:\n", "\n", " - **Action**: Check for partner airline options if necessary.\n", "\n", "- **Upgrade Availability**:\n", "\n", " - **Action**: Rebook without additional charge using **`rebook_without_fee(booking_reference)`** if only higher-class seats are available.\n", "\n", "### **6. Compensation and Refund Rules**\n", "\n", "#### **6.1 Refund Processing**\n", "\n", "- **Refund Timeline**:\n", "\n", " - **Action**: Inform the customer about the processing time.\n", "\n", "- **Refund Options**:\n", "\n", " - **Original Payment Method**:\n", "\n", " - **Action**: Default method.\n", "\n", " - **Travel Credits**:\n", "\n", " - **Action**: Offer by calling **`offer_flight_credit(booking_reference)`**.\n", "\n", "#### **6.2 Compensation**\n", "\n", "- **Eligibility Check**:\n", "\n", " - **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", "\n", "- **Meals and Accommodation**:\n", "\n", " - **Action**: If eligible, call **`offer_accommodation(booking_reference)`**.\n", "\n", "### **7. Special Cases**\n", "\n", "#### **7.1 Medical Emergencies**\n", "\n", "- **Flexible Cancellations**:\n", "\n", " - **Action**: Upon documentation, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "\n", "#### **7.2 Bereavement Cases**\n", "\n", "- **Special Flexibility**:\n", "\n", " - **Action**: With proper documentation, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "\n", "#### **7.3 Group Bookings**\n", "\n", "- **Partial Cancellations**:\n", "\n", " - **Action**: Call **`process_partial_group_cancellation(booking_reference)`**.\n", "\n", "- **Name Changes**:\n", "\n", " - **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "\n", "#### **7.4 Unaccompanied Minors**\n", "\n", "- **Rebooking with Supervision**:\n", "\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`**, ensuring supervision.\n", "\n", "- **Priority for Minors**:\n", "\n", " - **Action**: Prioritize by calling **`prioritize_missed_connections(booking_reference)`**.\n", "\n", "### **8. FAQs for Common Scenarios**\n", "\n", "**Q1: What if a customer’s connecting flight is affected?**\n", "\n", "- **Action**: Call **`prioritize_missed_connections(booking_reference)`**.\n", "\n", "**Q2: How do I handle customers seeking upgrades after a cancellation?**\n", "\n", "- **Action**: Use **`rebook_without_fee(booking_reference)`**.\n", "\n", "**Q3: What if a customer needs to change a destination?**\n", "\n", "- **Action**: Inform about fare differences and fees. Proceed by calling **`apply_change_fee(booking_reference)`**.\n", "\n", "**Q4: What are the rules around rebooking for third-party bookings?**\n", "\n", "- **Action**: Advise contacting the original booking agent. Proceed per policy if eligible.\n", "\n", "**Q5: What should I do if the customer requests compensation for inconvenience?**\n", "\n", "- **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", "\n", "**Q6: How do I handle a situation where a customer’s preferred flight is fully booked?**\n", "\n", "- **Action**: Call **`check_next_available_flight(booking_reference)`**.\n", "\n", "**Q7: What if a customer wants to change the name on the ticket?**\n", "\n", "- **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "\n", "**Q8: How should I proceed if the customer is unsure about accepting flight credit?**\n", "\n", "- **Action**: Offer a refund by calling **`process_full_refund(booking_reference)`** if policy allows.\n", "\n", "**Q9: What if the customer is a no-show due to unforeseen circumstances?**\n", "\n", "- **Action**: Assess the situation. If applicable, call **`offer_flight_credit(booking_reference)`**.\n", "\n", "**Q10: How do I handle requests for same-day flight changes?**\n", "\n", "- **Action**:\n", "\n", " - Call **`check_ticket_type(booking_reference)`**.\n", " - For **flexible tickets**, call **`process_change_no_fee(booking_reference)`**.\n", " - For **non-flexible tickets**, call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: Use \"change\" in all communications.\n", "\n", "**Q11: The customer has a flexible ticket and wants to change their flight without any fees. Which function should I call?**\n", "\n", "- **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", "- **Language**: Use \"change\".\n", "- **Do Not**: Use **`rebook_without_fee`** or the term \"rebook\".\n" ] }, { "data": { "text/markdown": [ "### Accuracy: 100.00%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>Your ticket is non-refundable. As per our policy for non-refundable tickets, I can offer you flight credit for future use. Would you like to proceed with that option?</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>True</td>\n", " <td>Great news! Your ticket is flexible, which means you can rebook your flight without any additional fees. Let's proceed with the rebooking process.</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 True \n", "8 True \n", "9 True \n", "10 True \n", "11 True \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", "16 True \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 Your ticket is non-refundable. As per our policy for non-refundable tickets, I can offer you flight credit for future use. Would you like to proceed with that option? \n", "4 None \n", "5 None \n", "6 None \n", "7 Great news! Your ticket is flexible, which means you can rebook your flight without any additional fees. Let's proceed with the rebooking process. \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: ### **3. Cancellations: Types and Policies**\n", "\n", "#### **3.1 Customer-Initiated Cancellations**\n", "\n", "- **Refundable Tickets**:\n", "\n", " - **Within 24 Hours of Booking**:\n", " - **Action**: Call **`process_full_refund(booking_reference)`**.\n", " - **Language**: \"I will process a full refund for your booking.\"\n", "\n", " - **Beyond 24 Hours**:\n", " - **Action Steps**:\n", " 1. Call **`check_ticket_type(booking_reference)`** to confirm refundable status.\n", " 2. If eligible, proceed with **`process_full_refund(booking_reference)`**.\n", " - **Language**: \"Your ticket is refundable. I will proceed with a full refund.\"\n", "\n", "- **Non-Refundable Tickets**:\n", "\n", " - **Action**: Offer flight credit by calling **`offer_flight_credit(booking_reference)`**.\n", " - **Language**: \"Your ticket is non-refundable. I can offer you flight credit for future travel.\"\n", "\n", "- **Flexible Tickets**:\n", "\n", " - **Action**: Offer options based on customer preference:\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", " - **Language**: \"You have a flexible ticket. Would you prefer a full refund or flight credit for future use?\"\n", "\n", "---\n", "\n", "### **4. Changes: Types and Policies**\n", "\n", "#### **4.1 Customer-Initiated Changes**\n", "\n", "- **Always**:\n", "\n", " - **Action**: Call **`check_ticket_type(booking_reference)`** before processing any changes.\n", " - **Language**: \"Let me check the type of ticket you have to assist you further.\"\n", "\n", "- **Same-Day Changes**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight today without any additional fees.\"\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight today. A change fee will apply.\"\n", "\n", "- **Changes in Advance**:\n", "\n", " - **Within 7 Days of Departure**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight. Please note that a change fee applies for changes within 7 days of departure.\"\n", "\n", " - **Beyond 7 Days**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight without any additional fees.\"\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight. A change fee will apply.\"\n", "\n", "#### **4.2 Airline-Initiated Changes**\n", "\n", "- **Schedule Changes**:\n", "\n", " - **Minor Changes (< 2 Hours)**:\n", " - **Action**: Inform the customer of the new schedule.\n", " - **Language**: \"Please note there's a minor schedule change to your flight.\"\n", "\n", " - **Major Changes (≥ 2 Hours)**:\n", " - **Action**: Offer options by calling:\n", " - **`rebook_without_fee(booking_reference)`**\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", " - **Language**: \"Your flight schedule has changed significantly. We can **rebook** you on an alternative flight, offer you a full refund, or provide flight credit.\"\n", "\n", "- **Airline-Initiated Disruptions**:\n", "\n", " - **Action**: Rebook the customer by calling **`rebook_without_fee(booking_reference)`**.\n", " - **Language**: \"Due to the disruption, we'll **rebook** you on the next available flight at no additional charge.\"\n", "\n", "---\n", "\n", "### **5. Rebooking Guidelines**\n", "\n", "- **For Airline-Initiated Actions Only**\n", "\n", " - **Priority Handling**:\n", " - **Action**: Use **`rebook_without_fee(booking_reference)`**.\n", " - **Language**: \"We are prioritizing your rebooking to minimize inconvenience.\"\n", "\n", " - **Interline Agreements**:\n", " - **Action**: Check partner airlines if no suitable flights are available.\n", " - **Language**: \"We are checking for available flights with our partner airlines to rebook you.\"\n", "\n", " - **Upgrade Availability**:\n", " - **Action**: If only higher-class seats are available, rebook without additional charge.\n", " - **Language**: \"An upgrade is available. We'll **rebook** you without any additional cost.\"\n", "\n", "---\n", "\n", "### **6. Compensation and Refund Rules**\n", "\n", "#### **6.1 Refund Processing**\n", "\n", "- **Refund Timeline**:\n", "\n", " - **Action**: Inform customers about processing times.\n", " - **Language**: \"Refunds are processed within 7-10 business days for credit cards and 14-20 business days for debit cards.\"\n", "\n", "- **Refund Options**:\n", "\n", " - **Original Payment Method**:\n", "\n", " - **Default Action**: Refund to the original payment method.\n", " - **Language**: \"The refund will be issued to your original payment method.\"\n", "\n", " - **Travel Credits**:\n", "\n", " - **Action**: Offer flight credit if preferred.\n", " - **Language**: \"Alternatively, you can choose to receive flight credit for future travel.\"\n", "\n", "#### **6.2 Compensation**\n", "\n", "- **Eligibility Check**:\n", "\n", " - **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", " - **Language**: \"I will check if you are eligible for any compensation.\"\n", "\n", "- **Meals and Accommodation**:\n", "\n", " - **Action**: If eligible, call **`offer_accommodation(booking_reference)`**.\n", " - **Language**: \"You are eligible for accommodation and meal vouchers during the delay.\"\n", "\n", "---\n", "\n", "### **7. Special Cases**\n", "\n", "#### **7.1 Medical Emergencies**\n", "\n", "- **Action**: Upon receiving a medical certificate, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "- **Language**: \"I'm sorry to hear about your situation. With the provided documentation, I can process a flexible cancellation.\"\n", "\n", "#### **7.2 Bereavement Cases**\n", "\n", "- **Action**: With proper documentation, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "- **Language**: \"My condolences for your loss. I can assist you with a flexible cancellation.\"\n", "\n", "#### **7.3 Group Bookings**\n", "\n", "- **Partial Cancellations**:\n", "\n", " - **Action**: Call **`process_partial_group_cancellation(booking_reference)`**.\n", " - **Language**: \"I can assist with canceling the booking for specific individuals in your group.\"\n", "\n", "- **Name Changes**:\n", "\n", " - **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", " - **Language**: \"I can process a name change on your booking.\"\n", "\n", "#### **7.4 Unaccompanied Minors**\n", "\n", "- **Rebooking with Supervision**:\n", "\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`**, ensuring supervision is arranged.\n", " - **Language**: \"We'll **rebook** your child and ensure supervision is in place.\"\n", "\n", "- **Priority for Minors**:\n", "\n", " - **Action**: Call **`prioritize_missed_connections(booking_reference)`**.\n", " - **Language**: \"We'll prioritize your child's connection to ensure safe and timely travel.\"\n", "\n", "---\n", "\n", "### **8. FAQs for Common Scenarios**\n", "\n", "**Q1: What if a customer’s connecting flight is affected?**\n", "\n", "- **Action**: Call **`prioritize_missed_connections(booking_reference)`**.\n", "- **Language**: \"We'll prioritize your rebooking to minimize delays.\"\n", "\n", "---\n", "\n", "**Q2: How do I handle customers seeking upgrades after a cancellation?**\n", "\n", "- **Action**: Use **`rebook_without_fee(booking_reference)`** for airline-initiated cancellations.\n", "- **Language**: \"We can **rebook** you and offer an upgrade at no extra cost.\"\n", "\n", "---\n", "\n", "**Q3: What if a customer needs to change a destination?**\n", "\n", "- **Action**: Inform about fare differences and fees. Call **`apply_change_fee(booking_reference)`**.\n", "- **Language**: \"You can **change** your destination. Additional fees or fare differences may apply.\"\n", "\n", "---\n", "\n", "**Q4: What are the rules around rebooking for third-party bookings?**\n", "\n", "- **Action**: Advise the customer to contact the original booking agent. Proceed per policy if eligible.\n", "- **Language**: \"Please contact your booking agent for changes. If needed, I can provide assistance per our policies.\"\n", "\n", "---\n", "\n", "**Q5: What should I do if the customer requests compensation for inconvenience?**\n", "\n", "- **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", "- **Language**: \"I'll check your eligibility for compensation due to the inconvenience.\"\n", "\n", "---\n", "\n", "**Q6: How do I handle a situation where a customer’s preferred flight is fully booked?**\n", "\n", "- **Action**: Call **`check_next_available_flight(booking_reference)`**.\n", "- **Language**: \"Your preferred flight is fully booked. I'll check the next available options for you.\"\n", "\n", "---\n", "\n", "**Q7: What if a customer wants to change the name on the ticket?**\n", "\n", "- **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "- **Language**: \"I can assist with changing the name on your ticket.\"\n", "\n", "---\n", "\n", "**Q8: How should I proceed if the customer is unsure about accepting flight credit?**\n", "\n", "- **Action**: Offer a refund by calling **`process_full_refund(booking_reference)`** if policy allows.\n", "- **Language**: \"If you prefer, I can process a full refund for you.\"\n", "\n", "---\n", "\n", "**Q9: What if the customer is a no-show due to unforeseen circumstances?**\n", "\n", "- **Action**: Assess the situation. If applicable, call **`offer_flight_credit(booking_reference)`**.\n", "- **Language**: \"I understand unforeseen circumstances occurred. I can offer you flight credit for future travel.\"\n", "\n", "---\n", "\n", "**Q10: How do I handle requests for same-day flight changes?**\n", "\n", "- **Action Steps**:\n", " 1. Call **`check_ticket_type(booking_reference)`**.\n", " 2. If the ticket is flexible, call **`process_change_no_fee(booking_reference)`**.\n", " 3. If the ticket is non-flexible, call **`apply_change_fee(booking_reference)`**.\n", "- **Language**:\n", " - For flexible tickets: \"You can **change** your flight today without any additional fees.\"\n", " - For non-flexible tickets: \"You can **change** your flight today. A change fee will apply.\"\n" ] }, { "data": { "text/markdown": [ "### Accuracy: 100.00%" ], "text/plain": [ "<IPython.core.display.Markdown object>" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>expected_function</th>\n", " <th>expected_inputs</th>\n", " <th>actual_function</th>\n", " <th>actual_inputs</th>\n", " <th>is_correct</th>\n", " <th>assistant_message_content</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'ABC123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'DEF456'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'GHI789'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'JKL012'}</td>\n", " <td>True</td>\n", " <td>Your ticket is non-refundable. However, I can offer you flight credit for future travel. Would you like to proceed with flight credit?</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>check_ticket_type</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'PQR678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>check_compensation_eligibility</td>\n", " <td>{'booking_reference': 'STU901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'VWX234'}</td>\n", " <td>True</td>\n", " <td>Great news! Your ticket is flexible, which means you can rebook your flight without any additional fees. Would you like to proceed with rebooking your flight?</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>process_flexible_cancellation</td>\n", " <td>{'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>permit_name_change</td>\n", " <td>{'booking_reference': 'BCD890'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>offer_accommodation</td>\n", " <td>{'booking_reference': 'EFG123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>process_change_no_fee</td>\n", " <td>{'booking_reference': 'GHI123'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>apply_change_fee</td>\n", " <td>{'booking_reference': 'JKL234'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>rebook_without_fee</td>\n", " <td>{'booking_reference': 'MNO345'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>offer_flight_credit</td>\n", " <td>{'booking_reference': 'PQR567'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>process_full_refund</td>\n", " <td>{'booking_reference': 'STU678'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>prioritize_missed_connections</td>\n", " <td>{'booking_reference': 'XYZ901'}</td>\n", " <td>True</td>\n", " <td>None</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " expected_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " expected_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " actual_function \\\n", "0 check_ticket_type \n", "1 check_ticket_type \n", "2 process_full_refund \n", "3 offer_flight_credit \n", "4 check_ticket_type \n", "5 prioritize_missed_connections \n", "6 check_compensation_eligibility \n", "7 process_change_no_fee \n", "8 process_flexible_cancellation \n", "9 permit_name_change \n", "10 offer_accommodation \n", "11 process_change_no_fee \n", "12 apply_change_fee \n", "13 rebook_without_fee \n", "14 offer_flight_credit \n", "15 process_full_refund \n", "16 prioritize_missed_connections \n", "\n", " actual_inputs \\\n", "0 {'booking_reference': 'ABC123'} \n", "1 {'booking_reference': 'DEF456'} \n", "2 {'booking_reference': 'GHI789'} \n", "3 {'booking_reference': 'JKL012'} \n", "4 {'booking_reference': 'MNO345'} \n", "5 {'booking_reference': 'PQR678'} \n", "6 {'booking_reference': 'STU901'} \n", "7 {'booking_reference': 'VWX234'} \n", "8 {'booking_reference': 'YZA567', 'medical_certificate': 'medical_certificate_001'} \n", "9 {'booking_reference': 'BCD890'} \n", "10 {'booking_reference': 'EFG123'} \n", "11 {'booking_reference': 'GHI123'} \n", "12 {'booking_reference': 'JKL234'} \n", "13 {'booking_reference': 'MNO345'} \n", "14 {'booking_reference': 'PQR567'} \n", "15 {'booking_reference': 'STU678'} \n", "16 {'booking_reference': 'XYZ901'} \n", "\n", " is_correct \\\n", "0 True \n", "1 True \n", "2 True \n", "3 True \n", "4 True \n", "5 True \n", "6 True \n", "7 True \n", "8 True \n", "9 True \n", "10 True \n", "11 True \n", "12 True \n", "13 True \n", "14 True \n", "15 True \n", "16 True \n", "\n", " assistant_message_content \n", "0 None \n", "1 None \n", "2 None \n", "3 Your ticket is non-refundable. However, I can offer you flight credit for future travel. Would you like to proceed with flight credit? \n", "4 None \n", "5 None \n", "6 None \n", "7 Great news! Your ticket is flexible, which means you can rebook your flight without any additional fees. Would you like to proceed with rebooking your flight? \n", "8 None \n", "9 None \n", "10 None \n", "11 None \n", "12 None \n", "13 None \n", "14 None \n", "15 None \n", "16 None " ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Corrected Policy: ## **Policies for Flight Cancellations and Changes**\n", "\n", "### **2. Important Clarifications**\n", "\n", "#### **2.1 Use of Terms in Customer Communications**\n", "\n", "- **\"Change\"**:\n", " - **Use When**: The **customer initiates** modifications to their booking (e.g., changing flight dates, times, destinations).\n", " - **Instructions**:\n", " - **Always use the term \"change\"** when assisting with customer-initiated requests.\n", " - **Do Not Use**: The term \"rebook\" in these scenarios.\n", " - **Examples**:\n", " - **Correct**: \"You can **change** your flight without any additional fees.\"\n", " - **Incorrect**: \"You can **rebook** your flight without any additional fees.\"\n", "\n", "- **\"Rebook\"**:\n", " - **Use When**: The **airline initiates** changes due to cancellations, significant delays, or disruptions.\n", " - **Instructions**:\n", " - Use the term \"**rebook**\" when the airline is adjusting the booking.\n", " - **Examples**:\n", " - \"We will **rebook** you on the next available flight at no additional charge.\"\n", "\n", "#### **2.2 Function Usage and Associated Language**\n", "\n", "- **Customer-Initiated Changes**:\n", " - **Functions**:\n", " - **`process_change_no_fee(booking_reference)`**\n", " - **`apply_change_fee(booking_reference)`**\n", " - **Language**:\n", " - **Use \"change\"** in all customer communications.\n", " - **Avoid** using \"rebook\".\n", "\n", "- **Airline-Initiated Changes**:\n", " - **Functions**:\n", " - **`rebook_without_fee(booking_reference)`**\n", " - **Language**:\n", " - **Use \"rebook\"** in customer communications.\n", "\n", "#### **2.3 Consistency in Communication**\n", "\n", "- **Ensure** that the language used in communications **matches the function** being called.\n", "- **Do Not** mix terms or use them interchangeably.\n", "- **Examples**:\n", " - **Correct**: \"You can **change** your flight...\" when using **`process_change_no_fee`**.\n", " - **Incorrect**: \"You can **rebook** your flight...\" when using **`process_change_no_fee`**.\n", "\n", "---\n", "\n", "### **3. Cancellations: Types and Policies**\n", "\n", "#### **3.1 Customer-Initiated Cancellations**\n", "\n", "- **Refundable Tickets**:\n", "\n", " - **Within 24 Hours of Booking**:\n", " - **Action**: Call **`process_full_refund(booking_reference)`**.\n", " - **Language**: \"I will process a full refund for your booking.\"\n", "\n", " - **Beyond 24 Hours**:\n", " - **Action Steps**:\n", " 1. Call **`check_ticket_type(booking_reference)`** to confirm refundable status.\n", " 2. If eligible, proceed with **`process_full_refund(booking_reference)`**.\n", " - **Language**: \"Your ticket is refundable. I will proceed with a full refund.\"\n", "\n", "- **Non-Refundable Tickets**:\n", " - **Action**: Offer flight credit by calling **`offer_flight_credit(booking_reference)`**.\n", " - **Language**: \"Your ticket is non-refundable. However, I can offer you flight credit for future travel.\"\n", "\n", "- **Flexible Tickets**:\n", " - **Action**: Offer options based on customer preference:\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", " - **Language**: \"You have a flexible ticket. Would you prefer a full refund or flight credit for future use?\"\n", "\n", "---\n", "\n", "### **4. Changes: Types and Policies**\n", "\n", "#### **4.1 Customer-Initiated Changes**\n", "\n", "- **Important**:\n", "\n", " - Always **call `check_ticket_type(booking_reference)` before processing any changes**.\n", " - **Use the term \"change\"** in all customer communications.\n", " - **Do Not Use**: The term \"rebook\" in these scenarios.\n", "\n", "- **Same-Day Changes**:\n", "\n", " - **Flexible Tickets**:\n", " - **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight today without any additional fees.\"\n", "\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight today. A change fee will apply.\"\n", "\n", "- **Changes in Advance**:\n", "\n", " - **Within 7 Days of Departure**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight. Please note that a change fee applies for changes within 7 days of departure.\"\n", "\n", " - **Beyond 7 Days**:\n", " - **Flexible Tickets**:\n", " - **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight without any additional fees.\"\n", " - **Non-Flexible Tickets**:\n", " - **Action**: Call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight. A change fee will apply.\"\n", "\n", "#### **4.2 Airline-Initiated Changes**\n", "\n", "- **Schedule Changes**:\n", "\n", " - **Minor Changes (< 2 Hours)**:\n", " - **Action**: Inform the customer of the new schedule.\n", " - **Language**: \"Please note there's a minor schedule change to your flight.\"\n", "\n", " - **Major Changes (≥ 2 Hours)**:\n", " - **Action**: Offer options by calling:\n", " - **`rebook_without_fee(booking_reference)`**\n", " - **`process_full_refund(booking_reference)`**\n", " - **`offer_flight_credit(booking_reference)`**\n", " - **Language**: \"Your flight schedule has changed significantly. We can **rebook** you on an alternative flight, offer you a full refund, or provide flight credit.\"\n", "\n", "- **Airline-Initiated Disruptions**:\n", " - **Action**: Rebook the customer by calling **`rebook_without_fee(booking_reference)`**.\n", " - **Language**: \"Due to the disruption, we'll **rebook** you on the next available flight at no additional charge.\"\n", "\n", "---\n", "\n", "### **5. Rebooking Guidelines**\n", "\n", "- **For Airline-Initiated Actions Only**\n", "\n", " - **Priority Handling**:\n", " - **Action**: Use **`rebook_without_fee(booking_reference)`**.\n", " - **Language**: \"We are prioritizing your rebooking to minimize inconvenience.\"\n", "\n", " - **Interline Agreements**:\n", " - **Action**: Check partner airlines if no suitable flights are available.\n", " - **Language**: \"We are checking for available flights with our partner airlines to rebook you.\"\n", "\n", " - **Upgrade Availability**:\n", " - **Action**: If only higher-class seats are available, rebook without additional charge.\n", " - **Language**: \"An upgrade is available. We'll **rebook** you without any additional cost.\"\n", "\n", "---\n", "\n", "### **6. Compensation and Refund Rules**\n", "\n", "#### **6.1 Refund Processing**\n", "\n", "- **Refund Timeline**:\n", " - **Action**: Inform customers about processing times.\n", " - **Language**: \"Refunds are processed within 7-10 business days for credit cards and 14-20 business days for debit cards.\"\n", "\n", "- **Refund Options**:\n", " - **Original Payment Method**:\n", " - **Default Action**: Refund to the original payment method.\n", " - **Language**: \"The refund will be issued to your original payment method.\"\n", " - **Travel Credits**:\n", " - **Action**: Offer flight credit if preferred.\n", " - **Language**: \"Alternatively, you can choose to receive flight credit for future travel.\"\n", "\n", "#### **6.2 Compensation**\n", "\n", "- **Eligibility Check**:\n", " - **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", " - **Language**: \"I will check if you are eligible for any compensation.\"\n", "\n", "- **Meals and Accommodation**:\n", " - **Action**: If eligible, call **`offer_accommodation(booking_reference)`**.\n", " - **Language**: \"You are eligible for accommodation and meal vouchers during the delay.\"\n", "\n", "---\n", "\n", "### **7. Special Cases**\n", "\n", "#### **7.1 Medical Emergencies**\n", "\n", "- **Action**: Upon receiving a medical certificate, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "- **Language**: \"I'm sorry to hear about your situation. With the provided documentation, I can process a flexible cancellation.\"\n", "\n", "#### **7.2 Bereavement Cases**\n", "\n", "- **Action**: With proper documentation, call **`process_flexible_cancellation(booking_reference, documentation)`**.\n", "- **Language**: \"My condolences for your loss. I can assist you with a flexible cancellation.\"\n", "\n", "#### **7.3 Group Bookings**\n", "\n", "- **Partial Cancellations**:\n", " - **Action**: Call **`process_partial_group_cancellation(booking_reference)`**.\n", " - **Language**: \"I can assist with canceling the booking for specific individuals in your group.\"\n", "\n", "- **Name Changes**:\n", " - **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", " - **Language**: \"I can process a name change on your booking.\"\n", "\n", "#### **7.4 Unaccompanied Minors**\n", "\n", "- **Rebooking with Supervision**:\n", " - **Action**: Call **`rebook_without_fee(booking_reference)`**, ensuring supervision is arranged.\n", " - **Language**: \"We'll **rebook** your child and ensure supervision is in place.\"\n", "\n", "- **Priority for Minors**:\n", " - **Action**: Call **`prioritize_missed_connections(booking_reference)`**.\n", " - **Language**: \"We'll prioritize your child's connection to ensure safe and timely travel.\"\n", "\n", "---\n", "\n", "### **8. FAQs for Common Scenarios**\n", "\n", "**Q1: What if a customer’s connecting flight is affected?**\n", "\n", "- **Action**: Call **`prioritize_missed_connections(booking_reference)`**.\n", "- **Language**: \"We'll prioritize your rebooking to minimize delays.\"\n", "\n", "---\n", "\n", "**Q2: How do I handle customers seeking upgrades after a cancellation?**\n", "\n", "- **Action**: Use **`rebook_without_fee(booking_reference)`** for airline-initiated cancellations.\n", "- **Language**: \"We can **rebook** you and offer an upgrade at no extra cost.\"\n", "\n", "---\n", "\n", "**Q3: What if a customer needs to change a destination?**\n", "\n", "- **Action**: Inform about fare differences and fees. Call **`apply_change_fee(booking_reference)`**.\n", "- **Language**: \"You can **change** your destination. Additional fees or fare differences may apply.\"\n", "\n", "---\n", "\n", "**Q4: What are the rules around rebooking for third-party bookings?**\n", "\n", "- **Action**: Advise the customer to contact the original booking agent. Proceed per policy if eligible.\n", "- **Language**: \"Please contact your booking agent for changes. If needed, I can provide assistance per our policies.\"\n", "\n", "---\n", "\n", "**Q5: What should I do if the customer requests compensation for inconvenience?**\n", "\n", "- **Action**: Call **`check_compensation_eligibility(booking_reference)`**.\n", "- **Language**: \"I'll check your eligibility for compensation due to the inconvenience.\"\n", "\n", "---\n", "\n", "**Q6: How do I handle a situation where a customer’s preferred flight is fully booked?**\n", "\n", "- **Action**: Call **`check_next_available_flight(booking_reference)`**.\n", "- **Language**: \"Your preferred flight is fully booked. I'll check the next available options for you.\"\n", "\n", "---\n", "\n", "**Q7: What if a customer wants to change the name on the ticket?**\n", "\n", "- **Action**: If requested 7+ days before departure, call **`permit_name_change(booking_reference)`**.\n", "- **Language**: \"I can assist with changing the name on your ticket.\"\n", "\n", "---\n", "\n", "**Q8: How should I proceed if the customer is unsure about accepting flight credit?**\n", "\n", "- **Action**: Offer a refund by calling **`process_full_refund(booking_reference)`** if policy allows.\n", "- **Language**: \"If you prefer, I can process a full refund for you.\"\n", "\n", "---\n", "\n", "**Q9: What if the customer is a no-show due to unforeseen circumstances?**\n", "\n", "- **Action**: Assess the situation. If applicable, call **`offer_flight_credit(booking_reference)`**.\n", "- **Language**: \"I understand unforeseen circumstances occurred. I can offer you flight credit for future travel.\"\n", "\n", "---\n", "\n", "**Q10: How do I handle requests for same-day flight changes?**\n", "\n", "- **Action Steps**:\n", " 1. Call **`check_ticket_type(booking_reference)`**.\n", " 2. If the ticket is flexible, call **`process_change_no_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight today without any additional fees.\"\n", " 3. If the ticket is non-flexible, call **`apply_change_fee(booking_reference)`**.\n", " - **Language**: \"You can **change** your flight today. A change fee will apply.\"\n", "\n", "---\n", "\n", "**Q11: The customer has a flexible ticket and wants to change their flight without any fees. Which function should I call?**\n", "\n", "- **Action**: Call **`process_change_no_fee(booking_reference)`**.\n", "- **Language**: \"You can **change** your flight without any additional fees.\"\n", "- **Do Not**: Use **`rebook_without_fee`** or the term \"rebook\" in this scenario.\n", "\n", "---\n", "\n", "**Note**: Always ensure that the **language used in communications aligns with the function being performed** and the **nature of the change** (customer-initiated vs. airline-initiated). This consistency helps avoid confusion and ensures adherence to company policies.\n" ] } ], "source": [ "messages = [\n", " {\n", " \"role\": \"user\",\n", " \"content\": f\"\"\"\n", "You are an agent that is responsible for improving the quality of routine instructions that are provided to a customer service LLM agent.\n", "\n", "I am going to give you the policy for the customer service agent that contains detailed instructions on how to handle flight cancellations and changes.\n", "\n", "You will also be provided with the results from an eval set that include the following:\n", " - conversation history: This is the conversation that we present to the LLM along with the system prompt\n", " - expected_function: This is the function we expect the LLM to call\n", " - expected_input: This is the input we expect the LLM to provide to the function\n", " - actual_function: This is the actual function the LLM called\n", " - actual_input: This is the actual input the LLM provided\n", " - assistant_message_content: This is the message the LLM generated when it returned its response\n", " - is_correct: True/False value depending on if the model responded correctly\n", "\n", "Carefully analyze the instructions provided as well as the results of the eval. Get a firm understanding of the failures in the policy.\n", "\n", "Return an updated policy that will perform better against the dataset.\n", "\n", "Here is the current policy:\n", "{updated_policy}\n", "\"\"\"\n", " }\n", "]\n", "\n", "for _ in range(5):\n", " # Evaluate the function calls with the current policy\n", " df, accuracy = evaluate_function_calls('evals/functionCallingEval.csv', updated_policy, 'gpt-3.5-turbo-0125')\n", " \n", " # Display the accuracy as a mini header\n", " display(Markdown(f\"### Accuracy: {accuracy:.2%}\"))\n", " display(df)\n", "\n", " results_json = df.to_json(orient='records')\n", "\n", " messages.append({\n", " \"role\": \"user\",\n", " \"content\": f\"\"\"\n", "Here are the results based on the current policy:\n", "{results_json}\n", "\"\"\"\n", " })\n", " # Use the metaprompt function to get an updated policy\n", " temp_policy_json = enforce_schema(metaprompt(messages))\n", " temp_policy_str = temp_policy_json.strip(\"json```\").strip(\"```\")\n", " temp_policy = json.loads(temp_policy_str)[\"final_answer\"]\n", " print(f\"Corrected Policy: {temp_policy}\")\n", "\n", " messages.append({\n", " \"role\": \"assistant\",\n", " \"content\": f\"\"\"\n", "{temp_policy}\n", "\"\"\"\n", " })\n", "\n", " # Update the policy for the next iteration\n", " updated_policy = temp_policy\n" ] } ], "metadata": { "kernelspec": { "display_name": "openai", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.8" } }, "nbformat": 4, "nbformat_minor": 2 }