{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with structured outputs\n",
    "\n",
    "If you've seen my [talk](https://www.youtube.com/watch?v=yj-wSRJwrrc&t=1s) on this topic, you can skip this chapter.\n",
    "\n",
    "tl;dr\n",
    "\n",
    "When we work with LLMs you find that many times we are not building chatbots, instead we're working with structured outputs in order to solve a problem by returning machine readable data. However the way we think about the problem is still very much influenced by the way we think about chatbots. This is a problem because it leads to a lot of confusion and frustration. In this chapter we'll try to understand why this happens and how we can fix it.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The fundamental problem with JSON and Dictionaries\n",
    "\n",
    "Lets say we have a simple JSON object, and we want to work with it. We can use the `json` module to load it into a dictionary, and then work with it. However, this is a bit of a pain, because we have to manually check the types of the data, and we have to manually check if the data is valid. For example, lets say we have a JSON object that looks like this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [{\"first_name\": \"Jason\", \"age\": 10}, {\"firstName\": \"Jason\", \"age\": \"10\"}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a `name` field, which is a string, and an `age` field, which is an integer. However, if we were to load this into a dictionary, we would have no way of knowing if the data is valid. For example, we could have a string for the age, or we could have a float for the age. We could also have a string for the name, or we could have a list for the name.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jason is 10\n",
      "None is 10\n",
      "Next year Jason will be 11 years old\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "can only concatenate str (not \"int\") to str",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb Cell 5\u001b[0m line \u001b[0;36m9\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb#W4sZmlsZQ%3D%3D?line=6'>7</a>\u001b[0m name \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mfirst_name\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb#W4sZmlsZQ%3D%3D?line=7'>8</a>\u001b[0m age \u001b[39m=\u001b[39m obj\u001b[39m.\u001b[39mget(\u001b[39m\"\u001b[39m\u001b[39mage\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb#W4sZmlsZQ%3D%3D?line=8'>9</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39mf\u001b[39m\u001b[39m\"\u001b[39m\u001b[39mNext year \u001b[39m\u001b[39m{\u001b[39;00mname\u001b[39m}\u001b[39;00m\u001b[39m will be \u001b[39m\u001b[39m{\u001b[39;00mage\u001b[39m+\u001b[39;49m\u001b[39m1\u001b[39;49m\u001b[39m}\u001b[39;00m\u001b[39m years old\u001b[39m\u001b[39m\"\u001b[39m)\n",
      "\u001b[0;31mTypeError\u001b[0m: can only concatenate str (not \"int\") to str"
     ]
    }
   ],
   "source": [
    "for obj in data:\n",
    "    name = obj.get(\"first_name\")\n",
    "    age = obj.get(\"age\")\n",
    "    print(f\"{name} is {age}\")\n",
    "\n",
    "for obj in data:\n",
    "    name = obj.get(\"first_name\")\n",
    "    age = obj.get(\"age\")\n",
    "    print(f\"Next year {name} will be {age+1} years old\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You see that while we were able to program with a dictionary, we had issues with the data being valid. We would have had to manually check the types of the data, and we had to manually check if the data was valid. This is a pain, and we can do better.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pydantic to the rescue\n",
    "\n",
    "Pydantic is a library that allows us to define data structures, and then validate them.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Person(name='Sam', age=30)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pydantic import BaseModel, Field\n",
    "\n",
    "\n",
    "class Person(BaseModel):\n",
    "    name: str\n",
    "    age: int\n",
    "\n",
    "\n",
    "person = Person(name=\"Sam\", age=30)\n",
    "person"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Person(name='Sam', age=30)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data is correctly casted to the right type\n",
    "person = Person.model_validate({\"name\": \"Sam\", \"age\": \"30\"})\n",
    "person"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "AssertionError",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb Cell 10\u001b[0m line \u001b[0;36m2\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb#X12sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mname \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mSam\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb#X12sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m \u001b[39massert\u001b[39;00m person\u001b[39m.\u001b[39mage \u001b[39m==\u001b[39m \u001b[39m20\u001b[39m\n",
      "\u001b[0;31mAssertionError\u001b[0m: "
     ]
    }
   ],
   "source": [
    "assert person.name == \"Sam\"\n",
    "assert person.age == 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValidationError",
     "evalue": "2 validation errors for Person\nname\n  Field required [type=missing, input_value={'first_name': 'Sam', 'age': '30.2'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.6/v/missing\nage\n  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='30.2', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.6/v/int_parsing",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValidationError\u001b[0m                           Traceback (most recent call last)",
      "\u001b[1;32m/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb Cell 11\u001b[0m line \u001b[0;36m2\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb#X13sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39m# Data is validated to get better error messages\u001b[39;00m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb#X13sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m person \u001b[39m=\u001b[39m Person\u001b[39m.\u001b[39;49mmodel_validate({\u001b[39m\"\u001b[39;49m\u001b[39mfirst_name\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39mSam\u001b[39;49m\u001b[39m\"\u001b[39;49m, \u001b[39m\"\u001b[39;49m\u001b[39mage\u001b[39;49m\u001b[39m\"\u001b[39;49m: \u001b[39m\"\u001b[39;49m\u001b[39m30.2\u001b[39;49m\u001b[39m\"\u001b[39;49m})\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/jasonliu/dev/instructor/docs/tutorials/1-introduction.ipynb#X13sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m person\n",
      "File \u001b[0;32m~/dev/instructor/.venv/lib/python3.11/site-packages/pydantic/main.py:509\u001b[0m, in \u001b[0;36mBaseModel.model_validate\u001b[0;34m(cls, obj, strict, from_attributes, context)\u001b[0m\n\u001b[1;32m    507\u001b[0m \u001b[39m# `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks\u001b[39;00m\n\u001b[1;32m    508\u001b[0m __tracebackhide__ \u001b[39m=\u001b[39m \u001b[39mTrue\u001b[39;00m\n\u001b[0;32m--> 509\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mcls\u001b[39;49m\u001b[39m.\u001b[39;49m__pydantic_validator__\u001b[39m.\u001b[39;49mvalidate_python(\n\u001b[1;32m    510\u001b[0m     obj, strict\u001b[39m=\u001b[39;49mstrict, from_attributes\u001b[39m=\u001b[39;49mfrom_attributes, context\u001b[39m=\u001b[39;49mcontext\n\u001b[1;32m    511\u001b[0m )\n",
      "\u001b[0;31mValidationError\u001b[0m: 2 validation errors for Person\nname\n  Field required [type=missing, input_value={'first_name': 'Sam', 'age': '30.2'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.6/v/missing\nage\n  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='30.2', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.6/v/int_parsing"
     ]
    }
   ],
   "source": [
    "# Data is validated to get better error messages\n",
    "person = Person.model_validate({\"first_name\": \"Sam\", \"age\": \"30.2\"})\n",
    "person"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By introducing pydantic into any python codebase you can get a lot of benefits. You can get type checking, you can get validation, and you can get autocomplete. This is a huge win, because it means you can catch errors before they happen. This is even more useful when we rely on language models to generate data for us.\n",
    "\n",
    "You can also define validators that are run on the data. This is useful because it means you can catch errors before they happen. For example, you can define a validator that checks if the age is greater than 0. This is useful because it means you can catch errors before they happen.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fundamental problem with asking for JSON from OpenAI\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "correctly parsed person=Person(name='Jason', age=10)\n",
      "correctly parsed person=Person(name='Jason', age=10)\n",
      "correctly parsed person=Person(name='Jason', age=10)\n",
      "correctly parsed person=Person(name='Jason', age=10)\n",
      "correctly parsed person=Person(name='Jason', age=10)\n",
      "correctly parsed person=Person(name='Jason', age=10)\n",
      "error!!\n",
      "{\"jason\": 10}\n",
      "correctly parsed person=Person(name='Jason', age=10)\n",
      "correctly parsed person=Person(name='Jason', age=10)\n",
      "correctly parsed person=Person(name='Jason', age=10)\n"
     ]
    }
   ],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI()\n",
    "\n",
    "resp = client.chat.completions.create(\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": \"Please give me jason is 10 as a json object ```json\\n\"},\n",
    "    ],\n",
    "    n=10,\n",
    "    temperature=1,\n",
    ")\n",
    "\n",
    "for choice in resp.choices:\n",
    "    json = choice.message.content\n",
    "    try:\n",
    "        person = Person.model_validate_json(json)\n",
    "        print(f\"correctly parsed {person=}\")\n",
    "    except Exception as e:\n",
    "        print(\"error!!\")\n",
    "        print(json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Function Calling\n",
    "\n",
    "The json could be anything! We could add more and more into a prompt and hope it works, or we can use something called [function calling](https://platform.openai.com/docs/guides/function-calling) to directly specify the schema we want.\n",
    "\n",
    "**Function Calling**\n",
    "\n",
    "In an API call, you can describe functions and have the model intelligently choose to output a JSON object containing arguments to call one or many functions. The Chat Completions API does not call the function; instead, the model generates JSON that you can use to call the function in your code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PersonBirthday(name='Jason Liu', age=30, birthday=datetime.date(2024, 2, 8))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import datetime\n",
    "\n",
    "\n",
    "class PersonBirthday(BaseModel):\n",
    "    name: str\n",
    "    age: int\n",
    "    birthday: datetime.date\n",
    "\n",
    "\n",
    "schema = {\n",
    "    \"properties\": {\n",
    "        \"name\": {\"type\": \"string\"},\n",
    "        \"age\": {\"type\": \"integer\"},\n",
    "        \"birthday\": {\"type\": \"string\", \"format\": \"YYYY-MM-DD\"},\n",
    "    },\n",
    "    \"required\": [\"name\", \"age\"],\n",
    "    \"type\": \"object\",\n",
    "}\n",
    "\n",
    "resp = client.chat.completions.create(\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": f\"Extract `Jason Liu is thirty years old his birthday is yesturday` into json today is {datetime.date.today()}\",\n",
    "        },\n",
    "    ],\n",
    "    functions=[{\"name\": \"Person\", \"parameters\": schema}],\n",
    "    function_call=\"auto\",\n",
    ")\n",
    "\n",
    "PersonBirthday.model_validate_json(resp.choices[0].message.function_call.arguments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But it turns out, pydantic actually not only does our serialization, we can define the schema as well as add additional documentation!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'properties': {'name': {'title': 'Name', 'type': 'string'},\n",
       "  'age': {'title': 'Age', 'type': 'integer'},\n",
       "  'birthday': {'format': 'date', 'title': 'Birthday', 'type': 'string'}},\n",
       " 'required': ['name', 'age', 'birthday'],\n",
       " 'title': 'PersonBirthday',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PersonBirthday.model_json_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even define nested complex schemas, and documentation with ease.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'$defs': {'Address': {'properties': {'address': {'description': 'Full street address',\n",
       "     'title': 'Address',\n",
       "     'type': 'string'},\n",
       "    'city': {'title': 'City', 'type': 'string'},\n",
       "    'state': {'title': 'State', 'type': 'string'}},\n",
       "   'required': ['address', 'city', 'state'],\n",
       "   'title': 'Address',\n",
       "   'type': 'object'}},\n",
       " 'description': 'A Person with an address',\n",
       " 'properties': {'name': {'title': 'Name', 'type': 'string'},\n",
       "  'age': {'title': 'Age', 'type': 'integer'},\n",
       "  'address': {'$ref': '#/$defs/Address'}},\n",
       " 'required': ['name', 'age', 'address'],\n",
       " 'title': 'PersonAddress',\n",
       " 'type': 'object'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Address(BaseModel):\n",
    "    address: str = Field(description=\"Full street address\")\n",
    "    city: str\n",
    "    state: str\n",
    "\n",
    "\n",
    "class PersonAddress(Person):\n",
    "    \"\"\"A Person with an address\"\"\"\n",
    "\n",
    "    address: Address\n",
    "\n",
    "\n",
    "PersonAddress.model_json_schema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These simple concepts become what we built into `instructor` and most of the work has been around documenting how we can leverage schema engineering.\n",
    "Except now we use `instructor.patch()` to add a bunch more capabilities to the OpenAI SDK.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The core idea around Instructor\n",
    "\n",
    "1. Using function calling allows us use a llm that is finetuned to use json_schema and output json.\n",
    "2. Pydantic can be used to define the object, schema, and validation in one single class, allow us to encapsulate everything neatly\n",
    "3. As a library with 100M downloads, we can leverage pydantic to do all the heavy lifting for us and fit nicely with the python ecosystem\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PersonAddress(name='Jason Liu', age=30, address=Address(address='123 Main St', city='San Francisco', state='CA'))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import instructor\n",
    "import datetime\n",
    "\n",
    "# patch the client to add `response_model` to the `create` method\n",
    "client = instructor.patch(OpenAI(), mode=instructor.Mode.MD_JSON)\n",
    "\n",
    "resp = client.chat.completions.create(\n",
    "    model=\"gpt-3.5-turbo-1106\",\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": f\"\"\"\n",
    "            Today is {datetime.date.today()} \n",
    "\n",
    "            Extract `Jason Liu is thirty years old his birthday is yesturday` \n",
    "            he lives at 123 Main St, San Francisco, CA\"\"\",\n",
    "        },\n",
    "    ],\n",
    "    response_model=PersonAddress,\n",
    ")\n",
    "resp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By defining `response_model` we can leverage pydantic to do all the heavy lifting. Later we'll introduce the other features that `instructor.patch()` adds to the OpenAI SDK.\n",
    "but for now, this small change allows us to do a lot more with the API.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is instructor the only way to do this?\n",
    "\n",
    "No. Libraries like Marvin, Langchain, and Llamaindex all now leverage the Pydantic object in similar ways. The goal is to be as light weight as possible, get you as close as possible to the openai api, and then get out of your way.\n",
    "\n",
    "More importantly, we've also added straight forward validation and reasking to the mix.\n",
    "\n",
    "The goal of instructor is to show you how to think about structured prompting and provide examples and documentation that you can take with you to any framework.\n",
    "\n",
    "For further exploration:\n",
    "\n",
    "- [Marvin](https://www.askmarvin.ai/)\n",
    "- [Langchain](https://python.langchain.com/docs/modules/model_io/output_parsers/pydantic)\n",
    "- [LlamaIndex](https://gpt-index.readthedocs.io/en/latest/examples/output_parsing/openai_pydantic_program.html)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
