docs

docs/examples/index.md

@@ -6,6 +6,7 @@ Welcome to the examples page. Here you will find emails that highlight a range o
 
 - [Classifying Text](classification.md)
 - [Extracting search requests into multiple search queries](search.md)
+- [Self Critique](self_critique.md)
 - [Exact citations using regex](exact_citations.md)
 - [One shot query planning](planning-tasks.md)
 - [Using recursive schema](recursive.md)
@@ -19,6 +20,7 @@ In this section, you will find examples demonstrating different aspects of our p
 
 - [Classfying Text](classification.md): Doing single and multi class prediction using enums.
 
+- [Self Critique](self_critique.md): Learn how to implement self-critique in AI systems using `llm_validator`
 
 - [Segmented Search](search.md): Learn how to perform segmented search using a multi task definition using function calling
 

docs/examples/self_critique.md

@@ -0,0 +1,141 @@
+# Self-Correction with `llm_validator`
+
+## Introduction
+
+This guide demonstrates how to use `llm_validator` for implementing self-healing. The objective is to showcase how an instructor can self-correct by using validation errors and helpful error messages.
+
+## Setup
+
+Import required modules and apply compatibility patches.
+
+```python
+from typing_extensions import Annotated
+from pydantic import BaseModel, BeforeValidator
+from instructor import llm_validator, patch
+import openai
+
+patch()
+```
+
+## Defining Models
+
+Before building validation logic, define a basic Pydantic model named `QuestionAnswer`.
+We'll use this model to generate a response without validation to see the output.
+
+```python
+class QuestionAnswer(BaseModel):
+    question: str
+    answer: str
+```
+
+## Generating a Response
+
+Here we coerce the model to generate a response that is objectionable.
+
+```python
+question = "What is the meaning of life?"
+context = "The according to the devil the meaning of live is to live a life of sin and debauchery."
+
+qa: QuestionAnswer = openai.ChatCompletion.create(
+    model="gpt-3.5-turbo",
+    response_model=QuestionAnswer,
+    messages=[
+        {
+            "role": "system",
+            "content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.",
+        },
+        {
+            "role": "user",
+            "content": f"using the context: {context}\n\nAnswer the following question: {question}",
+        },
+    ],
+)
+```
+
+### Output Before Validation
+
+While it calls out the objectionable content, it doesn't provide any details on how to correct it.
+
+```json
+{
+  "question": "What is the meaning of life?",
+  "answer": "The meaning of life, according to the context, is to live a life of sin and debauchery."
+}
+```
+
+## Adding Custom Validation
+
+By adding a validator to the `answer` field, we can try to catch the issue and correct it.
+Lets integrate `llm_validator` into the model and see the error message. Its important to not that you can use all of pydantic's validators as you would normally as long as you raise a `ValidationError` with a helpful error message as it will be used as part of the self correction prompt.
+
+```python
+class QuestionAnswerNoEvil(BaseModel):
+    question: str
+    answer: Annotated[
+        str,
+        BeforeValidator(
+            llm_validator("don't say objectionable things", allow_override=True)
+        ),
+    ]
+
+try:
+    qa: QuestionAnswerNoEvil = openai.ChatCompletion.create(
+        model="gpt-3.5-turbo",
+        response_model=QuestionAnswerNoEvil,
+        messages=[
+            {
+                "role": "system",
+                "content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.",
+            },
+            {
+                "role": "user",
+                "content": f"using the context: {context}\n\nAnswer the following question: {question}",
+            },
+        ],
+    )
+except Exception as e:
+    print(e)
+```
+
+### Output After Validation
+
+Now, we throw validation error that its objectionable and provide a helpful error message.
+
+```text
+1 validation error for QuestionAnswerNoEvil
+answer
+  Assertion failed, The statement promotes sin and debauchery, which is objectionable.
+```
+
+## Retrying with Corrections
+
+By adding the `max_retries` parameter, we can retry the request with corrections. and use the error message to correct the output.
+
+```python
+qa: QuestionAnswerNoEvil = openai.ChatCompletion.create(
+    model="gpt-3.5-turbo",
+    response_model=QuestionAnswerNoEvil,
+    max_retries=1,
+    messages=[
+        {
+            "role": "system",
+            "content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.",
+        },
+        {
+            "role": "user",
+            "content": f"using the context: {context}\n\nAnswer the following question: {question}",
+        },
+    ],
+)
+```
+
+### Final Output
+
+Now, we get a valid response that is not objectionable!
+
+```json
+{
+  "question": "What is the meaning of life?",
+  "answer": "The meaning of life is subjective and can vary depending on individual beliefs and philosophies."
+}
+```

mkdocs.yml

@@ -54,6 +54,7 @@ nav:
   - Use Cases:
       - 'Overview': 'examples/index.md'
       - 'Text Classification': 'examples/classification.md'
+      - 'Self-Critique Using Validators': 'examples/self_critique.md'
       - 'Exact Citations for RAG': 'examples/exact_citations.md'
       - 'Extracting Multiple Search Queries': 'examples/search.md'
       - 'One shot Query Planning': 'examples/planning-tasks.md'