diff --git a/docs/examples/index.md b/docs/examples/index.md index f657156..f03f66a 100644 --- a/docs/examples/index.md +++ b/docs/examples/index.md @@ -6,6 +6,7 @@ Welcome to the examples page. Here you will find emails that highlight a range o - [Classifying Text](classification.md) - [Extracting search requests into multiple search queries](search.md) +- [Self Critique](self_critique.md) - [Exact citations using regex](exact_citations.md) - [One shot query planning](planning-tasks.md) - [Using recursive schema](recursive.md) @@ -19,6 +20,7 @@ In this section, you will find examples demonstrating different aspects of our p - [Classfying Text](classification.md): Doing single and multi class prediction using enums. +- [Self Critique](self_critique.md): Learn how to implement self-critique in AI systems using `llm_validator` - [Segmented Search](search.md): Learn how to perform segmented search using a multi task definition using function calling diff --git a/docs/examples/self_critique.md b/docs/examples/self_critique.md new file mode 100644 index 0000000..0cfa552 --- /dev/null +++ b/docs/examples/self_critique.md @@ -0,0 +1,141 @@ +# Self-Correction with `llm_validator` + +## Introduction + +This guide demonstrates how to use `llm_validator` for implementing self-healing. The objective is to showcase how an instructor can self-correct by using validation errors and helpful error messages. + +## Setup + +Import required modules and apply compatibility patches. + +```python +from typing_extensions import Annotated +from pydantic import BaseModel, BeforeValidator +from instructor import llm_validator, patch +import openai + +patch() +``` + +## Defining Models + +Before building validation logic, define a basic Pydantic model named `QuestionAnswer`. +We'll use this model to generate a response without validation to see the output. + +```python +class QuestionAnswer(BaseModel): + question: str + answer: str +``` + +## Generating a Response + +Here we coerce the model to generate a response that is objectionable. + +```python +question = "What is the meaning of life?" +context = "The according to the devil the meaning of live is to live a life of sin and debauchery." + +qa: QuestionAnswer = openai.ChatCompletion.create( + model="gpt-3.5-turbo", + response_model=QuestionAnswer, + messages=[ + { + "role": "system", + "content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.", + }, + { + "role": "user", + "content": f"using the context: {context}\n\nAnswer the following question: {question}", + }, + ], +) +``` + +### Output Before Validation + +While it calls out the objectionable content, it doesn't provide any details on how to correct it. + +```json +{ + "question": "What is the meaning of life?", + "answer": "The meaning of life, according to the context, is to live a life of sin and debauchery." +} +``` + +## Adding Custom Validation + +By adding a validator to the `answer` field, we can try to catch the issue and correct it. +Lets integrate `llm_validator` into the model and see the error message. Its important to not that you can use all of pydantic's validators as you would normally as long as you raise a `ValidationError` with a helpful error message as it will be used as part of the self correction prompt. + +```python +class QuestionAnswerNoEvil(BaseModel): + question: str + answer: Annotated[ + str, + BeforeValidator( + llm_validator("don't say objectionable things", allow_override=True) + ), + ] + +try: + qa: QuestionAnswerNoEvil = openai.ChatCompletion.create( + model="gpt-3.5-turbo", + response_model=QuestionAnswerNoEvil, + messages=[ + { + "role": "system", + "content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.", + }, + { + "role": "user", + "content": f"using the context: {context}\n\nAnswer the following question: {question}", + }, + ], + ) +except Exception as e: + print(e) +``` + +### Output After Validation + +Now, we throw validation error that its objectionable and provide a helpful error message. + +```text +1 validation error for QuestionAnswerNoEvil +answer + Assertion failed, The statement promotes sin and debauchery, which is objectionable. +``` + +## Retrying with Corrections + +By adding the `max_retries` parameter, we can retry the request with corrections. and use the error message to correct the output. + +```python +qa: QuestionAnswerNoEvil = openai.ChatCompletion.create( + model="gpt-3.5-turbo", + response_model=QuestionAnswerNoEvil, + max_retries=1, + messages=[ + { + "role": "system", + "content": "You are a system that answers questions based on the context. answer exactly what the question asks using the context.", + }, + { + "role": "user", + "content": f"using the context: {context}\n\nAnswer the following question: {question}", + }, + ], +) +``` + +### Final Output + +Now, we get a valid response that is not objectionable! + +```json +{ + "question": "What is the meaning of life?", + "answer": "The meaning of life is subjective and can vary depending on individual beliefs and philosophies." +} +``` \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index a860fd2..d2017d0 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -54,6 +54,7 @@ nav: - Use Cases: - 'Overview': 'examples/index.md' - 'Text Classification': 'examples/classification.md' + - 'Self-Critique Using Validators': 'examples/self_critique.md' - 'Exact Citations for RAG': 'examples/exact_citations.md' - 'Extracting Multiple Search Queries': 'examples/search.md' - 'One shot Query Planning': 'examples/planning-tasks.md'