courses/agent_assist/baseline_summarization

{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "XvOluiKaG0Hb" }, "source": [ "# End to End Baseline Summarisation\n", "\n", "In this notebook you will use the configured conversation profile from earlier in the lab to perform summarization of audio transcripts with redacted PII. You will need the integration ID of your conversation profile created earlier to complete this lab.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "mLtMeOqgHVEc" }, "source": [ "# Installing required libraries and Authenticating GCP Credentials" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "mq2u5TNpGzHQ", "tags": [] }, "outputs": [], "source": [ "! pip install -q google-cloud-storage google-cloud-dlp google-cloud-dialogflow google-cloud-speech" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__IMPORTANT:__ You may ignore any error messages about the dependency resolver. After installing packages, restart the kernel for the notebook by going to __Kernel__ and __Restart Kernel__ before moving forward. You do not need to run the first cell again after completing the package installation" ] }, { "cell_type": "markdown", "metadata": { "id": "n6KG5mj1I-hr" }, "source": [ "## Configure Google Cloud credentials\n", "\n", "__Note:__ Replace `project-name` with your Project ID. You will need to uncomment the commented lines first if you are running this notebook in a Google Colab environment." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "cellView": "form", "id": "LF0QStZrHzkq", "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Updated property [core/project].\n" ] } ], "source": [ "PROJECT_NAME='your-project-id' \n", "\n", "!gcloud config set project $PROJECT_NAME" ] }, { "cell_type": "markdown", "metadata": { "id": "zgNLnpuPLK9T" }, "source": [ "## Import required libraries" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "aPhHyXo0LFSw", "tags": [] }, "outputs": [], "source": [ "from typing import Dict, List\n", "import csv\n", "import glob\n", "import json\n", "import time\n", "import re\n", "import json\n", "import pandas as pd\n", "import pickle\n", "from google.cloud import storage\n", "from google.cloud import speech_v1p1beta1 as speech\n", "import google.cloud.dlp\n", "from google.cloud import dialogflow_v2beta1 as dialogflow\n", "import datetime" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Replace the value of the`CONV_PROFILE_ID` variable with the integration ID you recorded earlier.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "cellView": "form", "id": "v5gWFyL2LSIe", "tags": [] }, "outputs": [], "source": [ "CONV_PROFILE_ID = \"projects/your-project-id/locations/global/conversationProfiles/conv-profile-id\"\n", "GCS_BUCKET_URI = \"gs://cloud-training\" \n", "GCS_BUCKET_NAME = GCS_BUCKET_URI.split(\"//\")[1]\n", "AUDIO_FILE_INPUT_FOLDER_PREFIX = \"specialized-training/ccai/audio_summarization\" \n", "SUPPORTED_FILE_FORMATS = [\"WAV\",\"MP3\"]\n", "\n", "NUM_CHANNELS = 1 # Audio file for this lab is single channel. Update for dual channel.\n", "LANGUAGE_CODE = \"en-US\" \n", "MODEL =\"phone_call\"\n", "SAMPLE_RATE_HZ = 8000 \n", "ENCODING = \"MULAW\" \n", "MIN_SPEAKER_COUNT = 2 \n", "MAX_SPEAKER_COUNT = 2 \n", "\n", "SPEECH_TO_TEXT_CONFIG = {\n", "\"audio_channel_count\" : NUM_CHANNELS,\n", "\"model\" : \"phone_call\",\n", "\"encoding\" : ENCODING,\n", "\"diarization_config\" : speech.SpeakerDiarizationConfig(enable_speaker_diarization=True,max_speaker_count=MAX_SPEAKER_COUNT,\n", " min_speaker_count=MIN_SPEAKER_COUNT),\n", "\"language_code\" : LANGUAGE_CODE,\n", "\"enableSeparateRecognitionPerChannel\" : True if NUM_CHANNELS>1 else False,\n", "\"sample_rate_hertz\" : SAMPLE_RATE_HZ\n", "}\n", "\n", "project_id = PROJECT_NAME\n", "location = \"global\"\n", "project_path = '/'.join(CONV_PROFILE_ID.split('/')[:4])\n", "conversation_profile_id = CONV_PROFILE_ID" ] }, { "cell_type": "markdown", "metadata": { "id": "g6vdll99TXgG" }, "source": [ "# Step 1: Transcribe conversation and run PII redaction on transcripts" ] }, { "cell_type": "markdown", "metadata": { "id": "NqEYIb8eTqYg" }, "source": [ "## Utility Functions" ] }, { "cell_type": "markdown", "metadata": { "id": "JyQA7VHUBW2C" }, "source": [ "Before summarizing transcripts, you will need to transcribe the audio to text and redact possibly sensitive information found in the transcripts. This will lower the risk of accidental data leakage.\n", "\n", "**Note**: `INFO_TYPES` should be fine-tuned to fit customer's requirements. The existing `INFO_TYPES` in the cell below is the default setting but is subject to developer's discretion. To fine-tune `INFO_TYPES`, please refer to https://cloud.google.com/dlp/docs/infotypes-reference\n", "\n", "First, instaniate a client to interact with the Data Loss Prevention (DLP) API and a function (`redact_dlp`) to redact sensitive information" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "id": "nEJ8d9aST8Cg", "tags": [] }, "outputs": [], "source": [ "dlp = google.cloud.dlp_v2.DlpServiceClient()\n", "INFO_TYPES = [\"AGE\",\"CREDIT_CARD_NUMBER\",\"CREDIT_CARD_TRACK_NUMBER\",\"DATE\",\"DATE_OF_BIRTH\",\n", " \"DOMAIN_NAME\",\"EMAIL_ADDRESS\",\"FEMALE_NAME\",\"MALE_NAME\",\"FIRST_NAME\",\"GENDER\",\n", " \"GENERIC_ID\",\"IP_ADDRESS\",\"LAST_NAME\",\"LOCATION\",\"PERSON_NAME\",\"PHONE_NUMBER\",\n", " \"STREET_ADDRESS\"]\n", "\n", "def redact_dlp(input_str,replacement_str=r\"[redacted]\"):\n", "\n", " inspect_config = {\"info_types\": [{\"name\": info_type} for info_type in INFO_TYPES]}\n", " deidentify_config = {\n", " \"info_type_transformations\": {\n", " \"transformations\": [\n", " {\n", " \"primitive_transformation\": {\n", " \"replace_config\": {\n", " \"new_value\": {\"string_value\": replacement_str}\n", " }\n", " }\n", " }\n", " ]\n", " }\n", " }\n", " item = {\"value\": input_str}\n", " response = dlp.deidentify_content(\n", " request={\n", " \"parent\" :\"projects/{}\".format(PROJECT_NAME),\n", " \"deidentify_config\": deidentify_config,\n", " \"inspect_config\": inspect_config,\n", " \"item\": item,\n", " }\n", " )\n", "\n", " return str(response.item.value).strip()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before defining a function to apply the DLP API, you define a function to transcribe your audio recording. The code following the definition of the `parse_transcript_results_single_channel` and `parse_transcript_results_dual_channel` functions imports the transcribed audio into a Pandas dataframe to make it easier to parse and apply the DLP API to the appropriate field of the transcripts." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "G4AKaXJgq0FM", "tags": [] }, "outputs": [], "source": [ "def is_agent_msg_text(text):\n", "\n", " text = text.lower()\n", "\n", " if 'thank you for calling' in text or 'monitored line' in text \\\n", " or 'recorded line' in text or 'monitor line' in text \\\n", " or 'how can i help you' in text or 'calling from' in text or 'may i ask' in text\\\n", " or 'advanced tech support' in text or 'support' in text:\n", "\n", " return True\n", "\n", " else:\n", "\n", " return False\n", "\n", "def parse_transcript_results_single_channel(file_name, results):\n", "\n", " words_info = results[-1].alternatives[0].words\n", " transcript_records_list = list()\n", " current_speaker_tag = words_info[0].speaker_tag\n", " agent_speaker_tag=words_info[0].speaker_tag #Generally, the first speaker is agent as observed by manually analysing transcripts in case channel is 1\n", " current_speaker_speech_text = \"\"\n", " turn = 0\n", "\n", " for _word in words_info:\n", "\n", " if _word.speaker_tag == current_speaker_tag:\n", "\n", " current_speaker_speech_text = current_speaker_speech_text + \" \" + _word.word\n", "\n", " else:\n", "\n", " transcript_records_list.append({ \"transcript_id\" : file_name,\"position\" : turn ,\n", " \"actor\":\"AGENT\" if current_speaker_tag==agent_speaker_tag else \"CUSTOMER\",\n", " \"speaker\" : current_speaker_tag,\n", " \"utterance\" : redact_dlp(current_speaker_speech_text.strip())})\n", " current_speaker_tag = _word.speaker_tag\n", " current_speaker_speech_text = \"\" + _word.word\n", " turn += 1\n", "\n", " return transcript_records_list\n", "\n", "# This function is not used in this lab, but recorded here for reference if working with dual channel audio.\n", "def parse_transcript_results_dual_channel(file_name, results):\n", "\n", " result_list=[]\n", " agent_channel=None\n", "\n", " for index,_result in enumerate(results):\n", "\n", " result_dict={}\n", " result_dict['transcript_id']=file_name\n", " _transcript =_result.alternatives[0].transcript\n", " _channel_tag=str(_result.channel_tag)\n", "\n", " if _transcript=='':\n", "\n", " continue\n", "\n", " result_dict['position']= index\n", "\n", " if agent_channel is None:\n", "\n", " if is_agent_msg_text(_transcript):\n", "\n", " if _channel_tag=='1':\n", "\n", " agent_channel='1'\n", "\n", " else:\n", "\n", " agent_channel='2'\n", "\n", " else:\n", "\n", " if _channel_tag=='1':\n", "\n", " agent_channel='2'\n", "\n", " else:\n", "\n", " agent_channel='1'\n", "\n", " result_dict['actor']= 'AGENT' if _channel_tag==agent_channel else 'CUSTOMER'\n", " result_dict['channel']=_channel_tag\n", " result_dict['utterance']= redact_dlp(_transcript.strip())\n", " result_list.append(result_dict)\n", "\n", " return result_list" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Audio Transcribed :: 1\n" ] } ], "source": [ "#Create clients for the Speech and Storage APIs\n", "\n", "speech_client = speech.SpeechClient()\n", "storage_client = storage.Client()\n", "\n", "# Define configuration for Speech API\n", "\n", "config = speech.RecognitionConfig(\n", " audio_channel_count= SPEECH_TO_TEXT_CONFIG[\"audio_channel_count\"],\n", " model= SPEECH_TO_TEXT_CONFIG[\"model\"],\n", " encoding=SPEECH_TO_TEXT_CONFIG[\"encoding\"],\n", " diarization_config= SPEECH_TO_TEXT_CONFIG[\"diarization_config\"],\n", " language_code= SPEECH_TO_TEXT_CONFIG[\"language_code\"],\n", " enable_separate_recognition_per_channel= SPEECH_TO_TEXT_CONFIG[\"enableSeparateRecognitionPerChannel\"],\n", " sample_rate_hertz=SPEECH_TO_TEXT_CONFIG[\"sample_rate_hertz\"],\n", " enable_automatic_punctuation= True,\n", " enable_word_time_offsets= True,\n", " enable_word_confidence=True,\n", " use_enhanced= True\n", " )\n", "\n", "INPUT_AUDIO_FILES_GCS_PATHS = storage_client.list_blobs(GCS_BUCKET_NAME, prefix= AUDIO_FILE_INPUT_FOLDER_PREFIX)\n", "index = 1\n", "all_transcripts = []\n", "_bucket = storage_client.get_bucket(GCS_BUCKET_NAME)\n", "\n", "# For each audio file do the following:\n", "# 1. Import the audio file\n", "# 2. Transcribe the conversation using the Speech API\n", "# 3. Parse the transcipts and redact sensitive information using the DLP API.\n", "\n", "for audio in INPUT_AUDIO_FILES_GCS_PATHS:\n", "\n", " if (str(audio.name).split(\"/\")[1] != '') and (str(audio.name).split(\"/\")[-1].split(\".\")[-1].upper() in SUPPORTED_FILE_FORMATS):\n", " \n", " try:\n", "\n", " audio_file = speech.RecognitionAudio(uri= GCS_BUCKET_URI + '/' + str(audio.name))\n", " response = speech_client.long_running_recognize(config=config, audio=audio_file)\n", " response_results = response.result()\n", "\n", " except Exception as e:\n", "\n", " print(\"Exception Occurred for Audio: {}\".format(e))\n", " continue\n", "\n", " if(len(response_results.results) != 0):\n", "\n", " temp=[]\n", "\n", " if SPEECH_TO_TEXT_CONFIG[\"audio_channel_count\"]==2:\n", "\n", " temp = parse_transcript_results_dual_channel(str(audio.name).split(\"/\")[1].split('.')[0],response_results.results)\n", "\n", " else:\n", "\n", " temp = parse_transcript_results_single_channel(str(audio.name).split(\"/\")[1].split('.')[0],response_results.results)\n", "\n", " all_transcripts.extend(temp)\n", " \n", " else:\n", " print('No files found.')\n", "\n", " print(f\"Audio Transcribed :: {str(index)}\")\n", " index += 1\n", " time.sleep(5) # Wait 5 sec per request to prevent \"ResourceExhausted\" error and avoid empty transcripts" ] }, { "cell_type": "markdown", "metadata": { "id": "thdjQUD2rodv" }, "source": [ "Before applying the baseline summarization model, you should explore the preprocessed and redacted output from one of the conversations. Here you will convert the `all_transcripts` into a Pandas dataframe and then look at one of the conversations. Note the portions of the conversation that were redacted by the DLP API." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>transcript_id</th>\n", " <th>position</th>\n", " <th>actor</th>\n", " <th>speaker</th>\n", " <th>utterance</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>ccai</td>\n", " <td>0</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>Hello. Thank you for contacting the customer s...</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>ccai</td>\n", " <td>1</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>to find out if there is a time slot for my exi...</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>ccai</td>\n", " <td>2</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>time. Sure. I can help you with that. I just n...</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>ccai</td>\n", " <td>3</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>a few things. Can I get your name?</td>\n", " </tr>\n", " <tr>\n", " <th>4</th>\n", " <td>ccai</td>\n", " <td>4</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>This is [redacted]</td>\n", " </tr>\n", " <tr>\n", " <th>5</th>\n", " <td>ccai</td>\n", " <td>5</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>and I'm showing a Yahoo email address on your ...</td>\n", " </tr>\n", " <tr>\n", " <th>6</th>\n", " <td>ccai</td>\n", " <td>6</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>yes, I am Shopper</td>\n", " </tr>\n", " <tr>\n", " <th>7</th>\n", " <td>ccai</td>\n", " <td>7</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>42 great. Do you happen to have a order number...</td>\n", " </tr>\n", " <tr>\n", " <th>8</th>\n", " <td>ccai</td>\n", " <td>8</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>If not, I can look it up. Let me</td>\n", " </tr>\n", " <tr>\n", " <th>9</th>\n", " <td>ccai</td>\n", " <td>9</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>grab it off the email I got with the order con...</td>\n", " </tr>\n", " <tr>\n", " <th>10</th>\n", " <td>ccai</td>\n", " <td>10</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>sure. Thank you. Of course. Okay. I</td>\n", " </tr>\n", " <tr>\n", " <th>11</th>\n", " <td>ccai</td>\n", " <td>11</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>see the order for a Samsung washer and dryer. ...</td>\n", " </tr>\n", " <tr>\n", " <th>12</th>\n", " <td>ccai</td>\n", " <td>12</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>easily that helps me plan for the day. Thank you</td>\n", " </tr>\n", " <tr>\n", " <th>13</th>\n", " <td>ccai</td>\n", " <td>13</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>so much for that information. I am happy I cou...</td>\n", " </tr>\n", " <tr>\n", " <th>14</th>\n", " <td>ccai</td>\n", " <td>14</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>[redacted] Yes, actually, can you tell me</td>\n", " </tr>\n", " <tr>\n", " <th>15</th>\n", " <td>ccai</td>\n", " <td>15</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>if I can have the electric dryer</td>\n", " </tr>\n", " <tr>\n", " <th>16</th>\n", " <td>ccai</td>\n", " <td>16</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>installed? Of course, let me pull that informa...</td>\n", " </tr>\n", " <tr>\n", " <th>17</th>\n", " <td>ccai</td>\n", " <td>17</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>up. I can see that we have an installation we ...</td>\n", " </tr>\n", " <tr>\n", " <th>18</th>\n", " <td>ccai</td>\n", " <td>18</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>well. Did you want to do that? Well, oh, no, t...</td>\n", " </tr>\n", " <tr>\n", " <th>19</th>\n", " <td>ccai</td>\n", " <td>19</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>okay. I can do an installation. But thank you ...</td>\n", " </tr>\n", " <tr>\n", " <th>20</th>\n", " <td>ccai</td>\n", " <td>20</td>\n", " <td>AGENT</td>\n", " <td>1</td>\n", " <td>for me sure as a reminder. You should receive ...</td>\n", " </tr>\n", " <tr>\n", " <th>21</th>\n", " <td>ccai</td>\n", " <td>21</td>\n", " <td>CUSTOMER</td>\n", " <td>2</td>\n", " <td>the evening prior, which would be [redacted] e...</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " transcript_id position actor speaker \\\n", "0 ccai 0 AGENT 1 \n", "1 ccai 1 CUSTOMER 2 \n", "2 ccai 2 AGENT 1 \n", "3 ccai 3 CUSTOMER 2 \n", "4 ccai 4 AGENT 1 \n", "5 ccai 5 CUSTOMER 2 \n", "6 ccai 6 AGENT 1 \n", "7 ccai 7 CUSTOMER 2 \n", "8 ccai 8 AGENT 1 \n", "9 ccai 9 CUSTOMER 2 \n", "10 ccai 10 AGENT 1 \n", "11 ccai 11 CUSTOMER 2 \n", "12 ccai 12 AGENT 1 \n", "13 ccai 13 CUSTOMER 2 \n", "14 ccai 14 AGENT 1 \n", "15 ccai 15 CUSTOMER 2 \n", "16 ccai 16 AGENT 1 \n", "17 ccai 17 CUSTOMER 2 \n", "18 ccai 18 AGENT 1 \n", "19 ccai 19 CUSTOMER 2 \n", "20 ccai 20 AGENT 1 \n", "21 ccai 21 CUSTOMER 2 \n", "\n", " utterance \n", "0 Hello. Thank you for contacting the customer s... \n", "1 to find out if there is a time slot for my exi... \n", "2 time. Sure. I can help you with that. I just n... \n", "3 a few things. Can I get your name? \n", "4 This is [redacted] \n", "5 and I'm showing a Yahoo email address on your ... \n", "6 yes, I am Shopper \n", "7 42 great. Do you happen to have a order number... \n", "8 If not, I can look it up. Let me \n", "9 grab it off the email I got with the order con... \n", "10 sure. Thank you. Of course. Okay. I \n", "11 see the order for a Samsung washer and dryer. ... \n", "12 easily that helps me plan for the day. Thank you \n", "13 so much for that information. I am happy I cou... \n", "14 [redacted] Yes, actually, can you tell me \n", "15 if I can have the electric dryer \n", "16 installed? Of course, let me pull that informa... \n", "17 up. I can see that we have an installation we ... \n", "18 well. Did you want to do that? Well, oh, no, t... \n", "19 okay. I can do an installation. But thank you ... \n", "20 for me sure as a reminder. You should receive ... \n", "21 the evening prior, which would be [redacted] e... " ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "eval_df = pd.DataFrame(all_transcripts)\n", "eval_df" ] }, { "cell_type": "markdown", "metadata": { "id": "c2j9vBCnXK17" }, "source": [ "# Step 2: Generate summaries from Baseline Summarization Model\n", "\n", "In this step you will generate summaries for the redacted transcripts from the previous steps after defining a sequence of helper functions to work through the appropriate steps. The comments in the code give a rough description of each of the helper functions being created." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "-BOSlFllaV2J", "tags": [] }, "outputs": [], "source": [ "# Function to create a conversation for a given conservation profile\n", "\n", "def create_conversation(client: dialogflow.ConversationsClient, project_id: str,\n", " conversation_profile_id: str):\n", "\n", " conversation = dialogflow.Conversation()\n", " conversation.conversation_profile = conversation_profile_id\n", "\n", " request = dialogflow.CreateConversationRequest(\n", " parent=project_id,\n", " conversation=conversation,\n", " )\n", " response = client.create_conversation(request=request)\n", " return response\n", "\n", "# Function to create a participant for a conversation (with a given conversation_id) with a specific role\n", "\n", "def create_participant(client: dialogflow.ParticipantsClient, conversation_id,\n", " role: dialogflow.Participant.Role):\n", "\n", " request = dialogflow.CreateParticipantRequest(\n", " parent=conversation_id,\n", " participant=dialogflow.Participant(role=role),\n", " )\n", " response = client.create_participant(request=request)\n", "\n", " return response\n", "\n", "# Function to suggest a conversation summary using the configured conversation profile.\n", "\n", "def suggest_conversation_summary(client: dialogflow.ConversationsClient,\n", " conversation_id: str):\n", "\n", " request = dialogflow.SuggestConversationSummaryRequest(\n", " conversation=conversation_id,)\n", " response = client.suggest_conversation_summary(request=request)\n", "\n", " return response\n", "\n", "# Function to complete a conversation with a given conversation id.\n", "\n", "def complete_conversation(client: dialogflow.ConversationsClient,\n", " conversation_id: str):\n", "\n", " request = dialogflow.CompleteConversationRequest(name=conversation_id,)\n", " response = client.complete_conversation(request)\n", "\n", " return response\n", "\n", "# Function to return a summary for a conversation using a specific conversation profile\n", "# using the earlier helper functions.\n", "\n", "def get_summary(\n", " conversations_client: dialogflow.ConversationsClient,\n", " participants_client: dialogflow.ParticipantsClient,\n", " project_id: str,\n", " conversation_profile_id: str,\n", " conversation,\n", "):\n", "\n", " create_conversation_response = create_conversation(\n", " client=conversations_client,\n", " project_id=project_id,\n", " conversation_profile_id=conversation_profile_id,\n", " )\n", " conversation_id = create_conversation_response.name\n", "\n", " create_end_user_participant_response = create_participant(\n", " client=participants_client,\n", " conversation_id=conversation_id,\n", " role=dialogflow.Participant.Role.END_USER,\n", " )\n", " end_user_participant_id = create_end_user_participant_response.name\n", "\n", " create_human_agent_participant_response = create_participant(\n", " client=participants_client,\n", " conversation_id=conversation_id,\n", " role=dialogflow.Participant.Role.HUMAN_AGENT,\n", " )\n", " human_agent_participant_id = create_human_agent_participant_response.name\n", "\n", " batch_request = dialogflow.BatchCreateMessagesRequest()\n", " batch_request.parent = conversation_id\n", " turn_count = 0\n", " for role, text in conversation:\n", " if turn_count > 199: # API was erroring out if the conv length is more than 200\n", " # Pushing first 200 messages into the conversation\n", " batch_response = conversations_client.batch_create_messages(request=batch_request)\n", "\n", " # re-initiatizing batch request to continue updating messages\n", " batch_request = dialogflow.BatchCreateMessagesRequest()\n", " batch_request.parent = conversation_id\n", "\n", " turn_count = 0\n", "\n", " participant_id = human_agent_participant_id if role == 'AGENT' else end_user_participant_id\n", "\n", " #Batch creating Conversation\n", " requests = dialogflow.CreateMessageRequest()\n", " requests.parent = conversation_id\n", " requests.message.content = text\n", " requests.message.participant = participant_id\n", " requests.message.send_time = datetime.datetime.now()\n", "\n", " batch_request.requests.append(requests)\n", " turn_count += 1\n", "\n", " batch_create_message_response = conversations_client.batch_create_messages(request=batch_request)\n", " suggest_conversation_summary_response = suggest_conversation_summary(\n", " client=conversations_client,\n", " conversation_id=conversation_id,\n", " )\n", "\n", " return suggest_conversation_summary_response" ] }, { "cell_type": "markdown", "metadata": { "id": "VHOXewKXs6KZ" }, "source": [ "Now call the Summarization API for transcript summarization to add the summary to the conversation strings." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "UG8Qi6zicZr2", "tags": [] }, "outputs": [], "source": [ "conversations_client = dialogflow.ConversationsClient()\n", "participants_client = dialogflow.ParticipantsClient()\n", "results = []\n", "\n", "for conversation_id in eval_df['transcript_id'].unique():\n", "\n", " #print(f'Running inference for: {conversation_id}')\n", " \n", " conversation = []\n", " conversation_df = eval_df.loc[(eval_df['transcript_id'] == conversation_id)]\n", "\n", " for idx in conversation_df.index:\n", "\n", " conversation.append((conversation_df.loc[idx, 'actor'], conversation_df.loc[idx, 'utterance']))\n", "\n", " get_summary_response = get_summary(\n", " conversations_client=conversations_client,\n", " participants_client=participants_client,\n", " project_id=project_path,\n", " conversation_profile_id=conversation_profile_id,\n", " conversation=conversation,\n", " )\n", "\n", " conversation_string = '\\n'.join(\n", " (f'{role}: {text}' for role, text in conversation))\n", " results.append({\n", " 'transcript_id': conversation_id,\n", " 'full_conversation': conversation_string,\n", " 'summary': get_summary_response.summary.text\n", " })\n" ] }, { "cell_type": "markdown", "metadata": { "id": "hlmNZLofrzeA" }, "source": [ "Now we can explore the output from the baseline summarization model for the conversation that you looked at earlier." ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "situation\n", "\n", "Customer wants to know if there is a time slot for their existing delivery. Customer also wants to know if they can have the electric dryer installed.\n", "\n", "action\n", "\n", "Agent checks the order status and informs the customer that the delivery is scheduled for [redacted] between 1 p.m. and 5 p.m. Agent also offers to schedule an installation for 69.99, but the customer declines.\n", "\n", "resolution\n", "\n", "Partial\n" ] } ], "source": [ "summ_df = pd.DataFrame(results)\n", "print(summ_df.iloc[0]['summary'].replace('\\n','\\n\\n'))" ] } ], "metadata": { "colab": { "provenance": [] }, "environment": { "kernel": "conda-root-py", "name": "workbench-notebooks.m115", "type": "gcloud", "uri": "gcr.io/deeplearning-platform-release/workbench-notebooks:m115" }, "kernelspec": { "display_name": "Python 3 (ipykernel) (Local) (Local)", "language": "python", "name": "conda-root-py" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 4 }

courses/agent_assist/baseline_summarization_audio.ipynb (965 lines of code) (raw):