{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5a01f3ac-5306-4a1b-9e47-a5d254bce93a",
   "metadata": {},
   "source": [
    "# Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dcc78ac-ed6d-49e3-b71b-fb2fb25f16a8",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "**What is a validation**\n",
    "\n",
    "Validation is the backbone of reliable software. Essentially, it involves checking whether the output of a function matches our expectations. Traditionally, validation was deterministic and rule-based, focusing on specific criteria (for example, 'output must not be larger than a certain value'). However, with the advent of Large Language Models (LLMs), we've seen the integration of probabilistic validation into our processes. This type of validation, often labeled as 'guardrail', might appear novel, but it's essentially another form of validation that leverages LLMs instead of strict rules to flag responses.\n",
    "\n",
    "In our approach at instructor, we treat all forms of validation equally. Whether it's rule-based, probabilistic, or a combination of both, everything is managed within a singular, cohesive framework. We achieve this through extensive use of Pydantic's powerful [validators](https://docs.pydantic.dev/latest/concepts/validators/#field-validators) feature."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "064c286b",
   "metadata": {},
   "source": [
    "Validators will enable us to control outputs by defining a function like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d4bb6258-b03a-4621-8a73-29056a20ec0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validation_function(value):\n",
    "    if condition(value):\n",
    "        raise ValueError(\"Value is not valid\")\n",
    "    return mutation(value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11d41e88-4629-4a44-a701-2bcdb297f090",
   "metadata": {},
   "source": [
    "The validation process in this framework unfolds in three key steps:\n",
    "\n",
    "* Condition Verification: The first step involves the validator checking whether a value meets a set condition. This is where the core validation logic is applied.\n",
    "\n",
    "* Error Handling with Optional Retry: If the value doesn't meet the condition, the system raises an error. There's an option to retry, offering a chance for correcting and reevaluating the value.\n",
    "\n",
    "* Value Processing: When the value meets the condition, the validator returns either the original or a modified version of the value. This ensures the output is valid and meets specific requirements or preferences."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "417fafe5-4616-4372-b9e9-78e89afff536",
   "metadata": {},
   "source": [
    "**Validation Applications**\n",
    "\n",
    "Validators are essential in tackling the unpredictabile nature of LLMs.\n",
    "\n",
    "Straightforward examples include:\n",
    "\n",
    "* Flagging outputs containing blacklisted words.\n",
    "* Identifying outputs with tones like racism or violence.\n",
    "\n",
    "For more complex tasks:\n",
    "\n",
    "* Ensuring citations directly come from provided content.\n",
    "* Checking that the model's responses align with given context.\n",
    "* Validating the syntax of SQL queries before execution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bd2104b-7eed-4619-a47d-c3d197f9d483",
   "metadata": {},
   "source": [
    "## Setup and Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e94449ab-50a9-4325-972c-f64fcdadee00",
   "metadata": {},
   "source": [
    "Using the [instructor](https://github.com/jxnl/instructor) library, we streamline the integration of these validators. `instructor` manages the parsing and validation of outputs and automates retries for compliant responses. This simplifies the process for developers to implement new validation logic, minimizing extra overhead."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7a84adc",
   "metadata": {},
   "source": [
    "To use instructor in our api calls, we just need to patch the openai client:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1aa2c503-82f8-4735-aae3-373b55fb1064",
   "metadata": {},
   "outputs": [],
   "source": [
    "import instructor \n",
    "from openai import OpenAI\n",
    "\n",
    "client = instructor.patch(OpenAI())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45cd244f-d59c-4431-be2d-aa356a6fefa0",
   "metadata": {},
   "source": [
    "## Software 2.0: Rule-based validators"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3494e664-c5b3-42ea-9c19-aa301a041bdb",
   "metadata": {},
   "source": [
    "Deterministic validation, characterized by its rule-based logic, ensures consistent outcomes for the same input. Let's explore how we can apply this concept through some examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "717ecefd-0355-4ba4-a642-95d281b0f075",
   "metadata": {},
   "source": [
    "### Flagging bad keywords"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a15013e-42f3-4d3b-b395-d6edbdec34e5",
   "metadata": {},
   "source": [
    "To begin with, we aim to prevent engagement in topics involving explicit violence."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13d61a81",
   "metadata": {},
   "source": [
    "We will define a blacklist of violent words that cannot be mentioned in any messages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "59330d7d-082a-4240-98c4-eaee18f02728",
   "metadata": {},
   "outputs": [],
   "source": [
    "blacklist = {\n",
    "    \"rob\",\n",
    "    \"steal\",\n",
    "    \"hurt\",\n",
    "    \"kill\",\n",
    "    \"attack\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ce06bbf",
   "metadata": {},
   "source": [
    "To validate if the message contains a blacklisted word we will use a [field_validator](https://jxnl.github.io/instructor/blog/2023/10/23/good-llm-validation-is-just-good-validation/#using-field_validator-decorator) over the 'message' field:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9bb87f47-db98-4f1d-80cb-ad5f39df8793",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 validation error for UserMessage\n",
      "message\n",
      "  Value error, `hurt` was found in the message `I will hurt him` [type=value_error, input_value='I will hurt him', input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.4/v/value_error\n"
     ]
    }
   ],
   "source": [
    "from pydantic import BaseModel, ValidationError, field_validator\n",
    "from pydantic.fields import Field\n",
    "\n",
    "class UserMessage(BaseModel):\n",
    "    message: str\n",
    "\n",
    "    @field_validator('message')\n",
    "    def message_cannot_have_blacklisted_words(cls, v: str) -> str:\n",
    "        for word in v.split(): \n",
    "            if word.lower() in blacklist:\n",
    "                raise ValueError(f\"`{word}` was found in the message `{v}`\")\n",
    "        return v\n",
    "\n",
    "try:\n",
    "    UserMessage(message=\"I will hurt him\")\n",
    "except ValidationError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e3a638-c9c9-44cd-bcd0-ad1a39f448db",
   "metadata": {},
   "source": [
    "### Flagging using OpenAI Moderation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88d0b816-7ec8-42b0-9b91-c9aab382c960",
   "metadata": {},
   "source": [
    "To enhance our validation measures, we'll extend the scope to flag any answer that contains hateful content, harassment, or similar issues. OpenAI offers a moderation endpoint that addresses these concerns, and it's freely available when using OpenAI models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65f46eb5",
   "metadata": {},
   "source": [
    "With the `instructor` library, this is just one function edit away:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "82521112-5301-4442-acce-82b495bd838f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class UserMessage(BaseModel):\n",
    "    message: str\n",
    "\n",
    "    @field_validator('message')\n",
    "    def message_must_comply_with_openai_mod(cls, v: str) -> str:\n",
    "        response = client.moderations.create(input=v)\n",
    "        out = response.results[0]\n",
    "        cats = dict(out.categories)\n",
    "        if out.flagged:\n",
    "            raise ValueError(f\"`{v}` was flagged for {[i for i in cats if cats[i]]}\")\n",
    "        \n",
    "        return v "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90542190-a4f2-4242-8261-2f0ace323022",
   "metadata": {},
   "source": [
    "Now we have a more comprehensive flagging for violence and we can outsource the moderation of our messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "54a9de1b-c6e7-4a5f-854c-506083a06a9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 validation error for UserMessage\n",
      "message\n",
      "  Value error, `I want to make them suffer the consequences` was flagged for ['harassment', 'harassment_threatening', 'violence', 'harassment/threatening'] [type=value_error, input_value='I want to make them suffer the consequences', input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.4/v/value_error\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    UserMessage(message=\"I want to make them suffer the consequences\")\n",
    "except ValidationError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f138f9f8-495a-4a09-96a0-c71d01561855",
   "metadata": {},
   "source": [
    "And as an extra, we get flagging for other topics like religion, race etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "feb77670-afd7-4947-89f8-a9446f6fb12c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 validation error for UserMessage\n",
      "message\n",
      "  Value error, `I will mock their religion` was flagged for ['harassment'] [type=value_error, input_value='I will mock their religion', input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.4/v/value_error\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    UserMessage(message=\"I will mock their religion\")\n",
    "except ValidationError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886f122b-22c9-440e-99cf-2e594b3df99b",
   "metadata": {},
   "source": [
    "### Filtering very long messages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "692b1164-4bd5-4943-b9ab-2edec00d4f7d",
   "metadata": {},
   "source": [
    "In addition to content-based flags, we can also set criteria based on other aspects of the input text. For instance, to maintain user engagement, we might want to prevent the assistant from returning excessively long texts. \n",
    "\n",
    "We can implement this using `instructor` to set a maximum word or character limit on the assistant's responses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "45ffdbd4-deae-4a46-9637-1b5339904f53",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AssistantMessage(BaseModel):\n",
    "    message: str\n",
    "\n",
    "    @field_validator('message')\n",
    "    def message_must_be_short(cls, v: str) -> str:\n",
    "        if len(v.split())>=100:\n",
    "            raise ValueError(f\"Text was flagged for being longer than 100 words.\")\n",
    "        \n",
    "        return v     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "66430dc5-b78c-45e2-a53b-ddc392b20583",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 validation error for AssistantMessage\n",
      "message\n",
      "  Value error, Text was flagged for being longer than 100 words. [type=value_error, input_value=\"\\n    Certainly! Lorem i... on the actual content.\", input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.4/v/value_error\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    AssistantMessage(message=\"\"\"\n",
    "    Certainly! Lorem ipsum is a placeholder text commonly used in the printing and typesetting industry. Here's a sample of Lorem ipsum text:\n",
    "\n",
    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam euismod velit vel tellus tempor, non viverra eros iaculis. Sed vel nisl nec mauris bibendum tincidunt. Vestibulum sed libero euismod, eleifend tellus id, laoreet elit. Donec auctor arcu ac mi feugiat, vel lobortis justo efficitur. Fusce vel odio vitae justo varius dignissim. Integer sollicitudin mi a justo bibendum ultrices. Quisque id nisl a lectus venenatis luctus.\n",
    "\n",
    "Please note that Lorem ipsum text is a nonsensical Latin-like text used as a placeholder for content, and it has no specific meaning. It's often used in design and publishing to demonstrate the visual aspects of a document without focusing on the actual content.\"\"\"\n",
    "                    )\n",
    "except ValidationError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "050e72fe-4b13-4002-a1d0-94f7b88b784b",
   "metadata": {},
   "source": [
    "### Avoiding hallucination"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3f2869e-c8a3-4b93-82e7-55eb70930900",
   "metadata": {},
   "source": [
    "When incorporating external knowledge bases, it's crucial to ensure that the agent uses the provided context accurately and doesn't fabricate responses. Validators can be effectively used for this purpose. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f67f7f92",
   "metadata": {},
   "source": [
    "We can illustrate this with an example where we validate that a provided citation is actually included in the referenced text chunk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "638fc368-5cf7-4ae7-9d3f-efea1b84eec0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import ValidationInfo\n",
    "\n",
    "class AnswerWithCitation(BaseModel):\n",
    "    answer: str\n",
    "    citation: str\n",
    "\n",
    "    @field_validator('citation')\n",
    "    @classmethod\n",
    "    def citation_exists(cls, v: str, info: ValidationInfo): \n",
    "        context = info.context\n",
    "        if context:\n",
    "            context = context.get('text_chunk')\n",
    "            if v not in context:\n",
    "                raise ValueError(f\"Citation `{v}` not found in text chunks\")\n",
    "        return v"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3064b06b-7f85-40ec-8fe2-4fa2cce36585",
   "metadata": {},
   "source": [
    "When the model responds with information not present in the provided context, the validation process comes into play:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0f3030b6-e6cf-45bf-a366-12de996fea40",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 validation error for AnswerWithCitation\n",
      "citation\n",
      "  Value error, Citation `Blueberries contain high levels of protein` not found in text chunks [type=value_error, input_value='Blueberries contain high levels of protein', input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.4/v/value_error\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    AnswerWithCitation.model_validate(\n",
    "        {\"answer\": \"Blueberries are packed with protein\", \"citation\": \"Blueberries contain high levels of protein\"},\n",
    "        context={\"text_chunk\": \"Blueberries are very rich in antioxidants\"}, \n",
    "    )\n",
    "except ValidationError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e54533-3304-4fa0-9828-9591d5dcdefd",
   "metadata": {},
   "source": [
    "## Software 3.0: Probabilistic validators"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1907df5b-472f-45ac-9181-45235e3cd0c3",
   "metadata": {},
   "source": [
    "For scenarios requiring more nuanced validation than what rule-based methods offer, we turn to probabilistic validation. This approach utilizes LLMs as a core component of the validation workflow, enabling a more sophisticated assessment of outputs.\n",
    "\n",
    "The `instructor` library facilitates this with its `llm_validator` utility. By specifying the desired directive, we can employ LLMs for complex validation tasks. Let's explore some intriguing use cases that are made possible through LLMs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd43d4c3-930a-4b3a-aded-3ad7308454ba",
   "metadata": {},
   "source": [
    "### Keeping an agent on topic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a9d80c-755f-434e-be52-2f5b58142b8c",
   "metadata": {},
   "source": [
    "In the case of creating an agent focused on health improvement, providing answers and daily practice suggestions, it's vital to ensure that the agent strictly adheres to health-related topics. This is important because the knowledge base is limited to health topics, and venturing beyond this scope could lead to  fabricated ('hallucinated') responses.\n",
    "\n",
    "To achieve this focus, we'll employ a similar process to the one previously discussed, but with an important addition: integrating an LLM into our validator. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "546625ac",
   "metadata": {},
   "source": [
    "This LLM will be tasked with determining whether the agent's responses are exclusively related to health topics. For this, we will use the `llm_validator` from `instructor` like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8cf00cad-c4c0-49dd-9be5-fb02338a5a7f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 validation error for AssistantMessage\n",
      "message\n",
      "  Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value='I would suggest you to v...is very nice in winter.', input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.4/v/assertion_error\n"
     ]
    }
   ],
   "source": [
    "from typing import Annotated\n",
    "from pydantic.functional_validators import AfterValidator\n",
    "from instructor import llm_validator\n",
    "\n",
    "class AssistantMessage(BaseModel):\n",
    "    message: Annotated[str, AfterValidator(llm_validator(\"don't talk about any other topic except health best practices and topics\"))]\n",
    "\n",
    "try:\n",
    "    AssistantMessage(message=\"I would suggest you to visit Sicily as they say it is very nice in winter.\")\n",
    "except ValidationError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dce5a7a-024e-4742-a124-fe51973df5f2",
   "metadata": {},
   "source": [
    "Great! With these validations, the model will reliably stick to its knowledge base, ensuring accurate and topic-specific responses."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6ec4afa-0be7-469e-93c0-5c729a06d4fc",
   "metadata": {},
   "source": [
    "### Validating agent thinking with CoT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "424d915b-f332-48f3-a75e-6e1cd6d12075",
   "metadata": {},
   "source": [
    "Using probabilistic validation, we can also assess the agent's reasoning process to ensure it's logical before providing a response. With [chain of thought](https://learnprompting.org/docs/intermediate/chain_of_thought) prompting, the model is expected to think in steps and arrive at an answer following its logical progression. If there are errors in this logic, the final response may be incorrect."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79c95242-6517-4ce2-aa99-4437db658057",
   "metadata": {},
   "source": [
    "Here we will use Pydantic's [model_validator](https://docs.pydantic.dev/latest/concepts/validators/#model-validators) which allows us to apply validation over all the properties of the `AIResponse` at once.\n",
    "\n",
    "To achieve this, we'll create a `Validation` class that specifies the desired output format from our LLM call, similar to how `llm_validator` functioned in the earlier example.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5211816e-c0fa-462d-acd0-7ab5a8096eb6",
   "metadata": {},
   "source": [
    "This `Validation` class will be used to determine if the chain of thought in the model's response is valid. If it's not valid, the class will also provide an explanation as to why:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "65340b8c-2ea3-4457-a6d4-f0e652c317b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Optional\n",
    "\n",
    "class Validation(BaseModel):\n",
    "    is_valid: bool = Field(..., description=\"Whether the value is valid based on the rules\")\n",
    "    error_message: Optional[str] = Field(..., description=\"The error message if the value is not valid, to be used for re-asking the model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de2104f1",
   "metadata": {},
   "source": [
    "The function we will call will integrate an LLM and will ask it to determine whether the answer the model provided follows from the chain of thought: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "e9ab3804-6962-4a48-83da-1f8360d8379a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_chain_of_thought(values):\n",
    "    chain_of_thought = values[\"chain_of_thought\"]\n",
    "    answer = values[\"answer\"]\n",
    "    resp = client.chat.completions.create(\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"You are a validator. Determine if the value follows from the statement. If it is not, explain why.\",\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": f\"Verify that `{answer}` follows the chain of thought: {chain_of_thought}\",\n",
    "            },\n",
    "        ],\n",
    "        # this comes from client = instructor.patch(OpenAI())\n",
    "        response_model=Validation,\n",
    "    )\n",
    "    print(resp)\n",
    "    if not resp.is_valid:\n",
    "        raise ValueError(resp.error_message)\n",
    "    return values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b79b94cf-15c2-432b-b0d5-aad0c2997f91",
   "metadata": {},
   "source": [
    "The use of the 'before' argument in this context is significant. It means that the validator will receive the complete dictionary of inputs in their raw form, before any parsing by Pydantic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "fbc9887a-df0d-4a4b-9ef5-ea450701d85b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any\n",
    "from pydantic import model_validator\n",
    "\n",
    "class AIResponse(BaseModel):\n",
    "    chain_of_thought: str\n",
    "    answer: str\n",
    "\n",
    "    @model_validator(mode='before')\n",
    "    @classmethod\n",
    "    def chain_of_thought_makes_sense(cls, data: Any) -> Any:\n",
    "        # here we assume data is the dict representation of the model\n",
    "        # since we use 'before' mode.\n",
    "        return validate_chain_of_thought(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "a38f2b28-f5b9-4a44-bfe5-9735726ec57d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "is_valid=False error_message='The user has a broken leg which is not related to diabetes.'\n",
      "1 validation error for AIResponse\n",
      "  Value error, The user has a broken leg which is not related to diabetes. [type=value_error, input_value={'chain_of_thought': 'The...user has a broken leg.'}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.4/v/value_error\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    resp = AIResponse(\n",
    "        chain_of_thought=\"The user suffers from diabetes.\", answer=\"The user has a broken leg.\"\n",
    ")\n",
    "except ValidationError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bbbaa11-32d2-4772-bc31-18d1d6d6c919",
   "metadata": {},
   "source": [
    "## Using validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39e642d9-0d20-4231-a694-baa0ea03f147",
   "metadata": {},
   "source": [
    "Integrating these validation examples with the OpenAI API is streamlined using `instructor`. After patching the OpenAI client with `instructor`, you simply need to specify a `response_model` for your requests. This setup ensures that all the validation processes occur automatically.\n",
    "\n",
    "Additionally, you can set a maximum number of retries. When calling the OpenAI client, the system can re-attempt to generate a correct answer. It does this by resending the original query along with feedback on why the previous response was rejected, guiding the LLM towards a more accurate answer in subsequent attempts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "97f544e7-2552-465c-89a9-a4820f00d658",
   "metadata": {},
   "outputs": [],
   "source": [
    "class HealthAnswer(BaseModel):\n",
    "    answer: Annotated[str, AfterValidator(llm_validator(\"don't talk about any other topic except health best practices and topics\"))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "bbd8aff1-4ad7-49d2-87f0-47ef155192ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 validation error for HealthAnswer\n",
      "answer\n",
      "  Assertion failed, The statement is not related to health best practices or topics. [type=assertion_error, input_value=\"While there isn't a sing...aking a final decision.\", input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.4/v/assertion_error\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    model = client.chat.completions.create(\n",
    "    model=\"gpt-3.5-turbo\",\n",
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": \"Which is the best headphone brand for producing music?\"},\n",
    "    ],\n",
    "    response_model=HealthAnswer,\n",
    "    max_retries=2,\n",
    ")\n",
    "except ValidationError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0c07b8b-ba6d-4e5d-a26c-ba72ca7d4f22",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344c623a-9b3b-4134-92d4-ad4eb9bb5f9e",
   "metadata": {},
   "source": [
    "This guide showed how to use deterministic and probabilistic validation techniques with Large Language Models. We covered using instructor to set up validation processes for filtering content, maintaining context relevance, and checking model reasoning. These methods improve the performance of LLMs across various tasks.\n",
    "\n",
    "For those looking to delve deeper here's a to-do list to explore:\n",
    "\n",
    "1. **SQL Syntax Checker**: Create a validator to check SQL query syntax before execution.\n",
    "2. **Context-Based Response Validation**: Design a method to flag responses based on the model's own knowledge rather than the provided context.\n",
    "3. **PII Detection**: Implement a mechanism to identify and handle Personally Identifiable Information in responses, focusing on maintaining user privacy.\n",
    "4. **Targeted Rule-Based Filtering**: Develop filters to remove certain content types, like responses mentioning named entities.\n",
    "    \n",
    "Completing these tasks will help users gain practical skills in enhancing LLMs through advanced validation methods."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pampa-labs",
   "language": "python",
   "name": "pampa-labs"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}