10-distillation/itinerary.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trip Itinerary Maker\n",
    "\n",
    "In this demo we will create trip itineraries with different models and prompts. Then we will take the best results and distill them into `gpt-4o-mini`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "itinerary_prompt = \"\"\"\n",
    "Create a customized travel itinerary for a given destination and duration that includes sights, restaurants, and unique attractions.\n",
    "\n",
    "Please provide the destination and the number of days for the trip. Each day's itinerary should help create an unforgettable experience, combining famous landmarks, off-the-beaten-path sites, dining recommendations, and other notable attractions.\n",
    "\n",
    "# Steps\n",
    "\n",
    "1. **Introduction and Overview:** Start with a broad introduction highlighting the given destination, describing its key characteristics, and noting what makes it unique.\n",
    "2. **Daily Itinerary Breakdown:**\n",
    "   - Provide a detailed itinerary for each day.\n",
    "   - Include a balanced mix of famous sights, lesser-known attractions, and a range of experiences (e.g., cultural, natural, adventure, etc.).\n",
    "   - Mention opening hours and expected time for each activity.\n",
    "3. **Food Recommendations:** \n",
    "   - Specify particular restaurants or must-try dishes.\n",
    "   - Attempt to align the meal plans with the day’s activities.\n",
    "4. **Unique Activities**:\n",
    "   - Suggest activities that will make the trip unforgettable—such as thrilling adventures, cultural immersion, or little-known hidden gems.\n",
    "5. **Conclude Each Day:** \n",
    "   - End each day's itinerary with a relaxing activity, such as a rooftop view, beach sunset, or evening local show, to ensure each day is well-rounded.\n",
    "\n",
    "# Notes\n",
    "\n",
    "- Be mindful of pacing to ensure the itinerary is enjoyable without being exhausting.\n",
    "- Include tips for travelers, such as best times to visit certain places and hidden gems.\n",
    "- Include any optional activities if time allows or if travelers prefer flexibility.\n",
    "- Make sure to not repeat any activities.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train and test prompts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"Generate a 7-day itinerary for Tokyo, Japan, focusing on cultural experiences and local cuisine.\",\n",
    "    \"Create a 5-day travel plan for Paris, France, that includes must-see landmarks and hidden gems.\",\n",
    "    \"Develop a 4-day trip itinerary for New York City, USA, highlighting sightseeing, museums, and dining.\",\n",
    "    \"Provide a 3-day schedule for a trip to Sydney, Australia, emphasizing outdoor activities and beaches.\",\n",
    "    \"Design a 6-day holiday plan for Rome, Italy, incorporating historical sites and authentic Italian restaurants.\",\n",
    "    \"Craft a 4-day adventure itinerary for Queenstown, New Zealand, focusing on adventure sports and nature.\",\n",
    "    \"Formulate a 2-day travel plan for Barcelona, Spain, including art, architecture, and local markets.\",\n",
    "    \"Outline a 3-day itinerary for Vancouver, Canada, highlighting natural parks and urban attractions.\",\n",
    "    \"Prepare a 5-day trip schedule for Cape Town, South Africa, featuring wildlife, beaches, and cultural sites.\",\n",
    "    \"Construct a 2-day itinerary for Bangkok, Thailand, focusing on temples, street food, and markets.\",\n",
    "    \"Generate a 5-day travel plan for Dubai, UAE, emphasizing modern architecture and desert experiences.\",\n",
    "    \"Create a 7-day itinerary for London, UK, including historical landmarks, museums, and theater shows.\",\n",
    "    \"Develop a 4-day schedule for Singapore, highlighting gardens, shopping districts, and local cuisine.\",\n",
    "    \"Provide a 3-day trip plan for Amsterdam, Netherlands, incorporating canals, museums, and cycling tours.\",\n",
    "    \"Design a 6-day itinerary for Lisbon, Portugal, focused on coastal views, historic neighborhoods, and local cuisine.\",\n",
    "    \"Craft an 6-day travel schedule for Istanbul, Turkey, featuring historical sites and unique dining experiences.\",\n",
    "    \"Formulate a 7-day itinerary for Bali, Indonesia, emphasizing beaches, temples, and wellness activities.\",\n",
    "    \"Outline a 5-day trip plan for Prague, Czech Republic, including castles, museums, and nightlife.\",\n",
    "    \"Prepare a 3-day itinerary for Los Angeles, USA, highlighting entertainment, beaches, and dining.\",\n",
    "    \"Construct a 4-day travel schedule for Athens, Greece, focusing on ancient sites and Mediterranean cuisine.\",\n",
    "    \"Generate a 6-day itinerary for Marrakech, Morocco, highlighting markets, palaces, and desert excursions.\",\n",
    "    \"Create a 5-day travel plan for Vienna, Austria, emphasizing classical music, historic architecture, and local cafés.\",\n",
    "    \"Develop a 3-day trip itinerary for Seoul, South Korea, focusing on modern attractions, traditional palaces, and street food.\",\n",
    "    \"Provide a 4-day schedule for Toronto, Canada, highlighting museums, diverse neighborhoods, and waterfront activities.\",\n",
    "    \"Design a 2-day travel plan for Edinburgh, Scotland, including castle tours, local pubs, and historic streets.\",\n",
    "    \"Craft a 5-day adventure itinerary for Reykjavik, Iceland, focusing on nature, hot springs, and outdoor activities.\",\n",
    "    \"Formulate a 7-day travel schedule for Hong Kong, China, incorporating urban exploration, cultural sites, and culinary delights.\",\n",
    "    \"Outline a 4-day trip plan for Dublin, Ireland, highlighting literary spots, historic landmarks, and traditional music venues.\",\n",
    "    \"Prepare a 3-day itinerary for Mumbai, India, emphasizing Bollywood tours, local markets, and historical sites.\",\n",
    "    \"Construct a 5-day travel schedule for Toronto, Canada, focusing on cultural festivals, museums, and diverse cuisine.\",\n",
    "    \"Generate a 2-day itinerary for Helsinki, Finland, highlighting design districts, waterfront walks, and local cafes.\",\n",
    "    \"Create a 4-day travel plan for Seoul, South Korea, concentrating on technology hubs, traditional markets, and cuisine.\",\n",
    "    \"Develop a 6-day itinerary for Copenhagen, Denmark, emphasizing design, cycling tours, and waterfront attractions.\",\n",
    "    \"Provide a 3-day trip plan for Zurich, Switzerland, incorporating lakeside activities, museums, and scenic walks.\",\n",
    "    \"Design a 5-day itinerary for Budapest, Hungary, centered on thermal baths, historic sites, and river cruises.\",\n",
    "    \"Craft a 2-day travel schedule for Brussels, Belgium, including chocolate tours, historic squares, and local breweries.\",\n",
    "    \"Formulate a 7-day trip itinerary for Dubrovnik, Croatia, focusing on coastal views, historic walls, and local cuisine.\",\n",
    "    \"Outline a 4-day travel plan for Montreal, Canada, highlighting cultural festivals, historic districts, and local cuisine.\",\n",
    "    \"Generate a 5-day itinerary for Melbourne, Australia, focusing on arts, dining, and coastal walks.\",\n",
    "    \"Create a 3-day travel plan for Seville, Spain, emphasizing flamenco, historic palaces, and local cuisine.\",\n",
    "    \"Develop a 6-day trip itinerary for Kyoto, Japan, focusing on temples, gardens, and traditional tea ceremonies.\",\n",
    "    \"Provide a 4-day schedule for Buenos Aires, Argentina, highlighting tango shows, historic neighborhoods, and local cuisine.\",\n",
    "    \"Design a 2-day travel plan for Geneva, Switzerland, including lakeside activities, international museums, and local chocolates.\",\n",
    "    \"Craft a 5-day adventure itinerary for Nairobi, Kenya, focusing on wildlife safaris, national parks, and cultural experiences.\",\n",
    "    \"Formulate a 7-day travel schedule for Amsterdam, Netherlands, incorporating canal tours, museums, and cycling routes.\",\n",
    "    \"Outline a 4-day trip plan for Budapest, Hungary, highlighting thermal spas, historic architecture, and river cruises.\",\n",
    "    \"Prepare a 3-day itinerary for Helsinki, Finland, emphasizing design museums, waterfront walks, and local cuisine.\",\n",
    "    \"Construct a 5-day travel schedule for Kyoto, Japan, focusing on temples, traditional markets, and gardens.\",\n",
    "    \"Generate a 2-day itinerary for Reykjavik, Iceland, highlighting geothermal pools, scenic walks, and local cuisine.\",\n",
    "    \"Create a 4-day travel plan for Barcelona, Spain, concentrating on architecture, beaches, and local eateries.\",\n",
    "    \"Develop a 6-day itinerary for Florence, Italy, emphasizing art galleries, historic sites, and Tuscan cuisine.\",\n",
    "    \"Provide a 3-day trip plan for Seoul, South Korea, incorporating modern attractions, traditional markets, and street food.\",\n",
    "    \"Design a 5-day itinerary for Oslo, Norway, centered on fjord cruises, museums, and local seafood.\",\n",
    "    \"Craft a 2-day travel schedule for Lisbon, Portugal, including historic trams, coastal views, and local cuisine.\"\n",
    "]\n",
    "\n",
    "test_prompts = [\n",
    "    \"Generate a 4-day itinerary for Madrid, Spain, focusing on art museums, historic plazas, and local tapas.\",\n",
    "    \"Create a 5-day travel plan for Istanbul, Turkey, emphasizing bazaars, culinary tours, and historic mosques.\",\n",
    "    \"Develop a 3-day trip itinerary for Vancouver, Canada, highlighting biking trails, seafood restaurants, and Stanley Park.\",\n",
    "    \"Provide a 6-day schedule for Sydney, Australia, incorporating harbor tours, coastal walks, and cultural festivals.\",\n",
    "    \"Design a 2-day holiday plan for Amsterdam, Netherlands, including canal cruises, bicycle tours, and local cafés.\",\n",
    "    \"Craft a 7-day adventure itinerary for Banff, Canada, focusing on hiking, wildlife spotting, and mountain photography.\",\n",
    "    \"Formulate a 4-day travel plan for Munich, Germany, emphasizing beer gardens, historic sites, and local cuisine.\",\n",
    "    \"Outline a 5-day itinerary for Athens, Greece, highlighting ancient ruins, Mediterranean dishes, and vibrant neighborhoods.\",\n",
    "    \"Prepare a 3-day trip schedule for Vienna, Austria, incorporating classical concerts, coffee houses, and imperial palaces.\",\n",
    "    \"Construct a 6-day itinerary for Lisbon, Portugal, focusing on tram rides, coastal excursions, and fado music.\",\n",
    "    \"Generate a 2-day travel plan for Zurich, Switzerland, highlighting alpine excursions, chocolate tastings, and lakeside walks.\",\n",
    "    \"Create a 4-day itinerary for Prague, Czech Republic, centered on gothic architecture, river cruises, and local breweries.\",\n",
    "    \"Develop a 5-day trip itinerary for Barcelona, Spain, emphasizing Gaudí's landmarks, beaches, and tapas tours.\",\n",
    "    \"Provide a 3-day schedule for Copenhagen, Denmark, incorporating Tivoli Gardens, harbor tours, and Danish cuisine.\",\n",
    "    \"Design a 7-day travel plan for Dubrovnik, Croatia, focusing on historical tours, island hopping, and local seafood.\",\n",
    "    \"Craft a 4-day adventure itinerary for Reykjavik, Iceland, highlighting glacier walks, geothermal spas, and Northern Lights.\",\n",
    "    \"Formulate a 5-day travel schedule for Florence, Italy, emphasizing Renaissance art, Tuscan wineries, and historic sites.\",\n",
    "    \"Outline a 3-day itinerary for Edinburgh, Scotland, including castle visits, whisky tastings, and historic walks.\",\n",
    "    \"Prepare a 6-day trip plan for Nairobi, Kenya, focusing on national parks, cultural museums, and urban experiences.\",\n",
    "    \"Construct a 2-day travel schedule for Geneva, Switzerland, highlighting international organizations, lakeside activities, and museums.\",\n",
    "    \"Generate a 4-day itinerary for Buenos Aires, Argentina, incorporating tango classes, historic neighborhoods, and steak tours.\",\n",
    "    \"Create a 5-day travel plan for Montreal, Canada, emphasizing bilingual culture, festivals, and local gastronomy.\",\n",
    "    \"Develop a 3-day itinerary for Seville, Spain, focusing on historic cathedrals, flamenco shows, and tapas bars.\",\n",
    "    \"Provide a 7-day schedule for Hong Kong, China, including skyline tours, harbor cruises, and culinary explorations.\",\n",
    "    \"Design a 4-day adventure plan for Queenstown, New Zealand, highlighting bungee jumping, vineyard tours, and lake activities.\",\n",
    "    \"Craft a 5-day trip itinerary for Los Angeles, USA, emphasizing Hollywood tours, beach visits, and diverse dining options.\",\n",
    "    \"Formulate a 3-day travel schedule for Mumbai, India, incorporating Bollywood experiences, street food tours, and historical landmarks.\",\n",
    "    \"Outline a 6-day itinerary for Florence, Italy, focusing on art workshops, historic tours, and scenic walks.\",\n",
    "    \"Prepare a 4-day plan for Oslo, Norway, highlighting fjord tours, Viking museums, and local seafood dining.\",\n",
    "    \"Construct a 5-day itinerary for Copenhagen, Denmark, emphasizing cycling tours, design museums, and waterfront activities.\",\n",
    "    \"Generate a 2-day travel plan for Geneva, Switzerland, focusing on botanical gardens, lake cruises, and international museum visits.\",\n",
    "    \"Create a 4-day itinerary for Dublin, Ireland, incorporating literary tours, historic pubs, and coastal walks.\",\n",
    "    \"Develop a 5-day trip schedule for Budapest, Hungary, highlighting ruin bars, thermal baths, and Danube river cruises.\",\n",
    "    \"Provide a 3-day schedule for Zurich, Switzerland, focusing on art galleries, boat tours, and Swiss cuisine.\",\n",
    "    \"Design a 7-day adventure plan for Banff, Canada, emphasizing mountain hiking, wildlife photography, and lake activities.\",\n",
    "    \"Craft a 4-day itinerary for Munich, Germany, including beer festival visits, historic tours, and Bavarian cuisine.\",\n",
    "    \"Formulate a 5-day travel plan for Athens, Greece, focusing on archaeological sites, Mediterranean dining, and vibrant markets.\",\n",
    "    \"Outline a 3-day trip itinerary for Vienna, Austria, incorporating imperial palaces, coffee house experiences, and classical music concerts.\",\n",
    "    \"Prepare a 6-day schedule for Lisbon, Portugal, emphasizing coastal drives, historic tram rides, and local culinary delights.\",\n",
    "    \"Construct a 2-day travel plan for Amsterdam, Netherlands, highlighting art museums, canal walks, and bicycle tours.\",\n",
    "    \"Generate a 4-day itinerary for Prague, Czech Republic, focusing on castle tours, river cruises, and local beer tastings.\",\n",
    "    \"Create a 5-day trip schedule for Barcelona, Spain, emphasizing beachfront activities, architectural tours, and tapas experiences.\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run on train set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I don't have real-time weather data to provide current conditions. To find the current weather in Tokyo, I recommend checking a reliable weather website or app for the latest updates.\n"
     ]
    }
   ],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI()\n",
    "\n",
    "for prompt in prompts:\n",
    "    model = \"gpt-4o\"\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": itinerary_prompt},\n",
    "        {\"role\": \"user\", \"content\": prompt}\n",
    "    ]\n",
    "    metadata = {\"model\": model, \"project\": \"itinerary\", \"run\": \"1\"}\n",
    "    res = client.chat.completions.create(model=model, messages=messages, store=True, metadata=metadata)\n",
    "    print(res.choices[0].message.content)\n",
    "\n",
    "for prompt in prompts:\n",
    "    model = \"o1-preview\"\n",
    "    messages = [\n",
    "        {\"role\": \"user\", \"content\": prompt}\n",
    "    ]\n",
    "    metadata = {\"model\": model, \"project\": \"itinerary\", \"run\": \"1\"}\n",
    "    res = client.chat.completions.create(model=model, messages=messages, store=True, metadata=metadata)\n",
    "    print(res.choices[0].message.content)\n",
    "\n",
    "for prompt in prompts:\n",
    "    model = \"gpt-4o-mini\"\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": itinerary_prompt},\n",
    "        {\"role\": \"user\", \"content\": prompt}\n",
    "    ]\n",
    "    metadata = {\"model\": model, \"project\": \"itinerary\", \"run\": \"1\"}\n",
    "    res = client.chat.completions.create(model=model, messages=messages, store=True, metadata=metadata)\n",
    "    print(res.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run on test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for prompt in test_prompts:\n",
    "    model = \"ft:gpt-4o-mini-2024-07-18:openai-devday-team:itinerary-test:ASZ0s97c\"\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": itinerary_prompt},\n",
    "        {\"role\": \"user\", \"content\": prompt}\n",
    "    ]\n",
    "    metadata = {\"model\": model, \"project\": \"itinerary\", \"run\": \"1\"}\n",
    "    res = client.chat.completions.create(model=model, messages=messages, store=True, metadata=metadata)\n",
    "    print(res.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate cost and latency savings\n",
    "\n",
    "For cost, when switching from 4o to 4o-mini fine-tune we can expect a ~1/8 reduction in price. Inference for 4o is $2.5 1m/input tokens and $10 1m/output tokens, while 4o-mini ft is $0.3 1m/input tokens and $1.2 1m/output tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "latencies = []\n",
    "latencies_new_model = []\n",
    "for prompt in test_prompts[:5]:\n",
    "    # Measure latency for original model\n",
    "    start_time = time.time()\n",
    "    res = client.chat.completions.create(\n",
    "        model=\"gpt-4o\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": itinerary_prompt},\n",
    "            {\"role\": \"user\", \"content\": prompt}\n",
    "        ]\n",
    "    )\n",
    "    latency = time.time() - start_time\n",
    "    latencies.append(latency)\n",
    "    print(f\"Latency for gpt-4o prompt: {latency:.4f} seconds\")\n",
    "    \n",
    "    # Measure latency for new model\n",
    "    start_time = time.time()\n",
    "    res = client.chat.completions.create(\n",
    "        model=\"ft:gpt-4o-mini-2024-07-18:openai-devday-team:itinerary-test:ASZ0s97c\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": itinerary_prompt},\n",
    "            {\"role\": \"user\", \"content\": prompt}\n",
    "        ]\n",
    "    )\n",
    "    latency_new = time.time() - start_time\n",
    "    latencies_new_model.append(latency_new)\n",
    "    print(f\"Latency for ft prompt: {latency_new:.4f} seconds\")\n",
    "\n",
    "average_latency = sum(latencies) / len(latencies) if latencies else 0\n",
    "average_latency_new_model = sum(latencies_new_model) / len(latencies_new_model) if latencies_new_model else 0\n",
    "print(f\"Average latency for gpt-4o: {average_latency:.4f} seconds\")\n",
    "print(f\"Average latency for ft:gpt-4o-mini-2024-07-18:openai-devday-team:itinerary-test:ASZ0s97c: {average_latency_new_model:.4f} seconds\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}