mirror of
https://github.com/kennethreitz/instructor.git
synced 2026-06-05 22:50:18 +00:00
Blog post: Validation is Validation (#121)
Co-authored-by: Rémi Louf <remilouf@protonmail.com> Co-authored-by: Ivan Leo <ivanleomk@gmail.com>
This commit is contained in:
Jason Liu
+216
-111
@@ -10,82 +10,9 @@ tags:
|
||||
|
||||
# Good LLM Validation is Just Good Validation
|
||||
|
||||
In the world of AI, validation plays a crucial role in ensuring the quality and reliability of generated outputs. Traditional approaches involve manual rule checking, but advancements in AI, such as Constitutional AI, offer a self-correcting system where AI models perform the validation. Pydantic and Instructor are powerful tools that enable validation without introducing new standards or terms. This post explores how to achieve effective validation using Pydantic and Instructor.
|
||||
> What if your validation logic could learn and adapt like a human, but operate at the speed of software? This is the future of validation and it's already here.
|
||||
|
||||
## Software 1.0 Validation
|
||||
|
||||
Pydantic provides various validation methods based on well-established patterns. Field validation in Pydantic can be done using the `field_validator` decorator or [PEP 593](https://www.python.org/dev/peps/pep-0593/) variable annotations. The official Pydantic documentation provides detailed information on these validation methods, including field validators and class validators.
|
||||
|
||||
### Example: Validating that a name contains a space
|
||||
|
||||
To illustrate field validation, let's consider the example of validating whether a name contains a space. Pydantic offers two approaches for this validation: using the `field_validator` decorator or the `Annotated` function.
|
||||
|
||||
#### Using `field_validator` decorator
|
||||
|
||||
Here's an example of using the `field_validator` decorator to define a validator for the `name` field:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, ValidationError, field_validator
|
||||
|
||||
class UserModel(BaseModel):
|
||||
id: int
|
||||
name: str
|
||||
|
||||
@field_validator('name')
|
||||
def name_must_contain_space(cls, v: str) -> str:
|
||||
if ' ' not in v:
|
||||
raise ValueError('must contain a space')
|
||||
return v.title()
|
||||
|
||||
try:
|
||||
UserModel(id=1, name='jason')
|
||||
except ValidationError as e:
|
||||
print(e)
|
||||
```
|
||||
|
||||
The code snippet demonstrates the validation process by raising a `ValueError` if the provided name does not contain a space. In the given example, the validation fails for the name 'jason,' and the corresponding error message is displayed:
|
||||
|
||||
```
|
||||
1 validation error for UserModel
|
||||
name
|
||||
Value error, must contain a space [type=value_error, input_value='jason', input_type=str]
|
||||
```
|
||||
|
||||
#### Using `Annotated`
|
||||
|
||||
Alternatively, you can use the `Annotated` function to validate that a name has a space. Here's an example:
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, ValidationError
|
||||
from pydantic.fields import Field
|
||||
from typing import Annotated
|
||||
|
||||
def name_must_contain_space(v):
|
||||
if ' ' not in v:
|
||||
raise ValueError('must contain a space')
|
||||
return v
|
||||
|
||||
class UserModel(BaseModel):
|
||||
id: int = Field(..., gt=0, lt=100)
|
||||
name: Annotated[str, name_must_contain_space]
|
||||
|
||||
try:
|
||||
UserModel(id=1, name='jason')
|
||||
except ValidationError as e:
|
||||
print(e)
|
||||
```
|
||||
|
||||
This code snippet achieves the same validation result. If the provided name does not contain a space, a `ValueError` is raised, and the corresponding error message is displayed:
|
||||
|
||||
```
|
||||
1 validation error for UserModel
|
||||
name
|
||||
Value error, must contain a space [type=value_error, input_value='jason', input_type=str]
|
||||
```
|
||||
|
||||
Validation is a fundamental concept in software development, and it remains the same when applied to AI systems. Instead of introducing new terms and standards, existing programming concepts can be leveraged. For example, types can have additional constraints, ensuring they are not "an apology" or "a threat." The underlying principles of validation remain unchanged.
|
||||
|
||||
In essence, validation involves checking if a value satisfies a condition. If it does, the value is returned. If it doesn't, an error is raised. This concept is similar to the examples mentioned above, with the addition of a possible mutation step:
|
||||
Validation is the backbone of reliable software. But traditional methods are static, rule-based, and can't adapt to new challenges. This post looks at how to bring dynamic, machine learning-driven validation into your software stack using Python libraries like Pydantic and Instructor. We validate these outputs using a validation function which conforms to the structure seen below.
|
||||
|
||||
```python
|
||||
def validation_function(value):
|
||||
@@ -94,39 +21,173 @@ def validation_function(value):
|
||||
return mutation(value)
|
||||
```
|
||||
|
||||
With Pydantic, we can define new types powered by probabilistic models and use them as validators.
|
||||
## What is instructor?
|
||||
|
||||
`Instructor` is built to interact with openai's function call api from python code, with python structs / objects. It's designed to be intuitive, easy to use, but give great visibily in how we call openai. By definining prompts as pydantic objects we can build in validators 'for free' and have a clear separation of concerns between the prompt and the code that calls openai.
|
||||
|
||||
```python
|
||||
import openai
|
||||
import instructor
|
||||
from pydantic import BaseModel
|
||||
|
||||
# This enables response_model keyword
|
||||
# from openai.ChatCompletion.create
|
||||
instructor.patch()
|
||||
|
||||
class UserDetail(BaseModel):
|
||||
name: str
|
||||
age: int
|
||||
|
||||
|
||||
user: UserDetail = openai.ChatCompletion.create(
|
||||
model="gpt-3.5-turbo",
|
||||
response_model=UserDetail,
|
||||
messages=[
|
||||
{"role": "user", "content": "Extract Jason is 25 years old"},
|
||||
]
|
||||
)
|
||||
|
||||
assert user.name == "Jason"
|
||||
assert user.age == 25
|
||||
```
|
||||
|
||||
In this post, we'll explore how to evolve from static, rule-based validation methods to dynamic, machine learning-driven ones. You'll learn how Python libraries like Pydantic and Instructor can make this transition smooth, and how Large Language Models can bring adaptability and nuance to your validation logic. We'll also delve into advanced topics like 'Chain of Thought' in validation and the importance of contextual checks.
|
||||
|
||||
Let's examine how these approaches with a example. Imagine that you run a software company who wants to ensure you never serve hateful and racist content. This isn't an easy job since the language around these topics change very quickly and frequently.
|
||||
|
||||
## Software 1.0 Validation
|
||||
|
||||
A simple method could be to compile a list of different words that are often associated with hate speech. This isn't a new approach - a quick google will throw up long and lengthy lists of these words and datesets. For simplicity, let's assume that we've found that the words `Steal` and `Rob` are good predictors of hateful speech from our database. We can modify our validation structure above to accomodate this.
|
||||
|
||||
```python
|
||||
def message_cannot_have_blacklisted_words(value):
|
||||
for word in value.split():
|
||||
if word.lower() in {'rob','steal'}:
|
||||
raise ValueError(f"`{word}`` was found in the message `{value}`")
|
||||
return mutation(value)
|
||||
```
|
||||
|
||||
This will throw an error if we pass in a string like `Let's rob the bank!` or `We should steal from the supermarkets`.
|
||||
|
||||
We can improve on this approach by using Pydantic, which provides various validation methods based on well-established patterns. Field validation in Pydantic can be done using the `field_validator` decorator or [PEP 593](https://www.python.org/dev/peps/pep-0593/) variable annotations. The official Pydantic documentation provides detailed information on these validation methods, including field validators and class validators.
|
||||
|
||||
### Migrating to Pydantic
|
||||
|
||||
Pydantic offers two approaches for this validation: using the `field_validator` decorator or the `Annotated` hints.
|
||||
|
||||
#### Using `field_validator` decorator
|
||||
|
||||
We can use the `field_validator` decorator to define a validator for a field in Pydantic. Here's a quick example of how we might be able to do so.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, ValidationError,field_validator
|
||||
from pydantic.fields import Field
|
||||
|
||||
|
||||
class UserMessage(BaseModel):
|
||||
message: str
|
||||
|
||||
@field_validator('message')
|
||||
def message_cannot_have_blacklisted_words(cls, v: str) -> str:
|
||||
for word in v.split():
|
||||
if word.lower() in {'rob','steal'}:
|
||||
raise ValueError(f"`{word}` was found in the message `{v}`")
|
||||
return v
|
||||
|
||||
try:
|
||||
UserMessage(message="This is a lovely day")
|
||||
UserMessage(message="We should go and rob a bank")
|
||||
except ValidationError as e:
|
||||
print(e)
|
||||
```
|
||||
|
||||
Since the message `This is a lovely day` does not have any blacklisted words, no errors are thrown. However, in the given example above, the validation fails for the message `We should go and rob a bank` due to the presence of the word `rob` and the corresponding error message is displayed.
|
||||
|
||||
```
|
||||
1 validation error for UserMessage
|
||||
message
|
||||
Value error, `rob` was found in the message `We should go and rob a bank` [type=value_error, input_value='We should go and rob a bank', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.4/v/value_error
|
||||
```
|
||||
|
||||
#### Using `Annotated`
|
||||
|
||||
Alternatively, you can use the `Annotated` function to perform the same validation. Here's an example where we utilise the same function we started with.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, ValidationError
|
||||
from typing import Annotated
|
||||
from pydantic.functional_validators import AfterValidator
|
||||
|
||||
|
||||
def message_cannot_have_blacklisted_words(value:str):
|
||||
for word in value.split():
|
||||
if word.lower() in {'rob','steal'}:
|
||||
raise ValueError(f"`{word}` was found in the message `{value}`")
|
||||
return value
|
||||
|
||||
class UserMessage(BaseModel):
|
||||
message: Annotated[str, AfterValidator(message_cannot_have_blacklisted_words)]
|
||||
|
||||
try:
|
||||
UserMessage(message="This is a lovely day")
|
||||
UserMessage(message="We should go and rob a bank")
|
||||
except ValidationError as e:
|
||||
print(e)
|
||||
```
|
||||
|
||||
This code snippet achieves the same validation result. If the provided name does not contain a space, a `ValueError` is raised, and the corresponding error message is displayed:
|
||||
|
||||
```
|
||||
1 validation error for UserMessage
|
||||
message
|
||||
Value error, `rob` was found in the message `We should go and rob a bank` [type=value_error, input_value='We should go and rob a bank', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.4/v/value_error
|
||||
```
|
||||
|
||||
On a high level, `Annotated` allows us to define a specific type and its corresponding validation which allows for easy re-use and abstraction. Field Validators on the other hand, are specific to the class themselves. The decision to use one over the other is often a matter of preference and how we want to manage our codebase.
|
||||
|
||||
Validation is a fundamental concept in software development and remains the same when applied to AI systems. Existing programming concepts should be leveraged when possible instead of introducing new terms and standards. The underlying principles of validation remain unchanged.
|
||||
|
||||
## Software 3.0: Validation for LLMs or powered by LLMs
|
||||
|
||||
Building upon the understanding of simple field validators, let's delve into probabilistic validation in software 3.0. In this context, we introduce an LLM-powered validator called `llm_validator` that uses a statement to verify the value. The model evaluates the statement to determine if the value is valid. If it is, the model returns the value; otherwise, it returns an error message.
|
||||
Now that we've understood how to use simple field validators, let's delve into probablistic validation. Building upon the understanding of simple field validators, let's delve into probabilistic validation in software 2.0. In this context, we introduce an LLM-powered validator called `llm_validator` that uses a statement to verify the value. The model evaluates the statement to determine if the value is valid. If it is, the model returns the value; otherwise, it returns an error message.
|
||||
|
||||
### Example: Don't Say Objectionable Things
|
||||
### Where Software 1.0 fails
|
||||
|
||||
Suppose we want to validate that a user's beliefs do not contain objectionable content. We can use the `llm_validator` to achieve this. Here's an example:
|
||||
Suppose now that we've gotten a new message - `Violence is always acceptable, as long as we silence the witness`. Our original validator wouldn't throw any errors when passed this new message since it uses neither the words `rob` or `steal`. However, it's clear that it is not a message which should be published.
|
||||
|
||||
We can get around this by using the inbuilt `llm_validator` class from `instructor`.
|
||||
|
||||
```python
|
||||
from instructor import llm_validator
|
||||
from pydantic import BaseModel, ValidationError
|
||||
from typing import Annotated
|
||||
from pydantic.functional_validators import AfterValidator
|
||||
|
||||
class UserModel(BaseModel):
|
||||
id: int
|
||||
name: str
|
||||
beliefs: Annotated[str, llm_validator("don't say objectionable things")]
|
||||
```
|
||||
import openai
|
||||
|
||||
Now, if we create a `UserModel` instance with a belief that contains objectionable content, we will receive an error.
|
||||
openai.api_key = # Input your open ai key here
|
||||
|
||||
class UserMessage(BaseModel):
|
||||
message: Annotated[str, AfterValidator(llm_validator("don't say objectionable things"))]
|
||||
|
||||
```python
|
||||
try:
|
||||
UserModel(id=1, name="Jason Liu", beliefs="We should steal from the poor")
|
||||
UserMessage(message="Violence is always acceptable, as long as we silence the witness")
|
||||
except ValidationError as e:
|
||||
print(e)
|
||||
print(e)
|
||||
```
|
||||
|
||||
The error message is generated by the language model (LLM) rather than the code itself, making it helpful for re-asking the model. Multiple validators can be stacked on top of each other.
|
||||
This produces the following error message as seen below
|
||||
|
||||
To better understand this approach, let's see how to build an `llm_validator` from scratch.
|
||||
```
|
||||
1 validation error for UserMessage
|
||||
message
|
||||
Assertion failed, The statement promotes violence, which is objectionable. [type=assertion_error, input_value='Violence is always accep... we silence the witness', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.4/v/assertion_error
|
||||
```
|
||||
|
||||
The error message is generated by the language model (LLM) rather than the code itself, making it helpful for re-asking the model. Multiple validators can be stacked on top of each other. To better understand this approach, let's see how to build an `llm_validator` from scratch.
|
||||
|
||||
### Creating Your Own Field Level `llm_validator`
|
||||
|
||||
@@ -138,7 +199,7 @@ Before we continue, let's review the anatomy of a validator:
|
||||
def validation_function(value):
|
||||
if condition(value):
|
||||
raise ValueError("Value is not valid")
|
||||
return value
|
||||
return value
|
||||
```
|
||||
|
||||
As we can see, a validator is simply a function that takes in a value and returns a value. If the value is not valid, it raises a `ValueError`. We can represent this using the following structure:
|
||||
@@ -152,7 +213,7 @@ class Validation(BaseModel):
|
||||
Using this structure, we can implement the same logic as before and utilize `instructor` to generate the validation.
|
||||
|
||||
```python
|
||||
import instructor
|
||||
import instructor
|
||||
import openai
|
||||
|
||||
# Enables `response_model` and `max_retries` parameters
|
||||
@@ -174,7 +235,7 @@ def validator(v):
|
||||
],
|
||||
# this comes from instructor.patch()
|
||||
response_model=Validation,
|
||||
)
|
||||
)
|
||||
if not resp.is_valid:
|
||||
raise ValueError(resp.error_message)
|
||||
return v
|
||||
@@ -186,25 +247,45 @@ Now we can use this validator in the same way we used the `llm_validator` from `
|
||||
from pydantic import BaseModel, ValidationError, field_validator, AfterValidator
|
||||
from typing import Annotated
|
||||
|
||||
class UserModel(BaseModel):
|
||||
id: int
|
||||
name: str
|
||||
beliefs: Annotated[str, AfterValidator(validator)]
|
||||
class UserMessage(BaseModel):
|
||||
message: Annotated[str, AfterValidator(validator)]
|
||||
```
|
||||
|
||||
## Writing validations that depend on multiple fields
|
||||
## Writing more complex validations
|
||||
|
||||
To validate multiple attributes simultaneously, you can extend the validation function and use a model validator instead of a field validator. Here's an example implementation in Python that checks if the `answer` follows the `chain_of_thought`:
|
||||
### Chain Of Thought
|
||||
|
||||
A popular way of prompting large language models nowadays is known as chain of thought. This involves getting a model to generate reasons and explanations for an answer to a prompt.
|
||||
|
||||
For instance, if we asked it the question
|
||||
|
||||
> If Will has 10 apples and James takes 4, how many apples does Will have?
|
||||
|
||||
A normal response would just be to output the response
|
||||
|
||||
> Will has 6 apples left
|
||||
|
||||
However, we can modify our prompt to utilise chain of thought prompting as
|
||||
|
||||
> If Will has 10 apples and james takes 4, how many apples does Will have? Let's think step by step.
|
||||
|
||||
This will cause it to output a more detailed response such as
|
||||
|
||||
> If Will has 10 apples and James takes 4, this means that Will will have less than 10 apples. If Will gives 4 apples to James, then this means that we should subtract 4 from 10. This leaves us with a final answer of 6. Therefore Will has 6 apples left.
|
||||
|
||||
Notice how the answer is significantly more detailed with explicit reasoning provided for the final response. We can utilise pydantic and instructor to perform a similar validation. Except in our case, instead of prompting a LLM to generate a chain of thought explanation, we'll be getting it to determine if a conclusion can be derived from a list of given reasons.
|
||||
|
||||
#### Implementation
|
||||
|
||||
One simple method is to extend our validation functions and utilise a model validator instead of a field validor. This allows us to perform a validation using a subset of all the fields in the model. Here's an example implementation in Python that checks if a `answer` folllows the `chain_of_thought`.
|
||||
|
||||
```python
|
||||
import instructor
|
||||
import instructor
|
||||
import openai
|
||||
|
||||
# Enables `response_model` and `max_retries` parameters
|
||||
instructor.patch()
|
||||
|
||||
# We assume a validator on a model takes in the dict
|
||||
# that comes in before other validation
|
||||
def validate_chain_of_thought(values):
|
||||
chain_of_thought = values["chain_of_thought"]
|
||||
answer = values["answer"]
|
||||
@@ -222,13 +303,15 @@ def validate_chain_of_thought(values):
|
||||
],
|
||||
# this comes from instructor.patch()
|
||||
response_model=Validation,
|
||||
)
|
||||
)
|
||||
if not resp.is_valid:
|
||||
raise ValueError(resp.error_message)
|
||||
return values
|
||||
```
|
||||
|
||||
To define a model validator, use the `@model_validator` decorator:
|
||||
We can then take advantage of the `model_validator` decorator to perform a validation on a subset of the model's data.
|
||||
|
||||
> We're defining a model validator here which runs before pydantic parses the input into its respective fields. That's why we have a **before** keyword used in the `model_validator` class.
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, model_validator
|
||||
@@ -260,13 +343,31 @@ If we create a `Response` instance with an answer that does not follow the chain
|
||||
|
||||
```
|
||||
1 validation error for Response
|
||||
Value error, The statement 'The meaning of life is 42' does not follow the chain of thought: 1 + 1 = 2.
|
||||
Value error, The statement 'The meaning of life is 42' does not follow the chain of thought: 1 + 1 = 2.
|
||||
[type=value_error, input_value={'chain_of_thought': '1 +... meaning of life is 42'}, input_type=dict]
|
||||
```
|
||||
|
||||
## Example: Citations, allowing Context to Influence Validation
|
||||
Beyong validating multiple attributes of a model, we can also introduce context to our validation functions, in order to give our models more information to work with.
|
||||
|
||||
Contextual information can be passed to validation methods by using a context object, which can be accessed from the `info` argument in decorated validator functions. This technique allows the model to validate text in the context of other text chunks. Here's an example:
|
||||
### Validating Citations From Original Text
|
||||
|
||||
Let's see a more concrete example. Let's say that we have the following answer
|
||||
|
||||
> Jason is a cool guy
|
||||
|
||||
a piece of text where it's supposed to have come from
|
||||
|
||||
> Jason is cool
|
||||
|
||||
and a original paragraph that we want to evaluate this against
|
||||
|
||||
> Jason is just a guy
|
||||
|
||||
#### Pydantic Context
|
||||
|
||||
How can we ensure that our citations support our answers with respect to an original source text? Well, Pydantic allows us to do so easily by utilising a context object. This is an arbitrary dictionary which you can access inside the `info` argument in a decorated validator function.
|
||||
|
||||
However, in order to do so, we need to utilise the `model_validate` function instead of creating classes as we've been doing so above. We can see a simplified example below.
|
||||
|
||||
```python
|
||||
class AnswerWithCitation(BaseModel):
|
||||
@@ -284,7 +385,7 @@ class AnswerWithCitation(BaseModel):
|
||||
return v
|
||||
```
|
||||
|
||||
Suppose you have a model with the following text chunks:
|
||||
We can then take our original example and test it against our new model
|
||||
|
||||
```python
|
||||
try:
|
||||
@@ -296,6 +397,8 @@ except ValidationError as e:
|
||||
print(e)
|
||||
```
|
||||
|
||||
This in turn generates the following error since `Jason is cool` does not exist in the text `Jason is just a guy`.
|
||||
|
||||
```
|
||||
1 validation error for AnswerWithCitation
|
||||
citation
|
||||
@@ -303,6 +406,8 @@ Value error, Citation `Jason is cool` not found in text chunks [type=value_error
|
||||
For further information visit https://errors.pydantic.dev/2.4/v/value_error
|
||||
```
|
||||
|
||||
#### Using Instructor.patch()
|
||||
|
||||
To pass this context from the `openai.ChatCompletion.create` call, `instructor.patch()` also passes the `validation_context`, which will be accessible from the `info` argument in the decorated validator functions.
|
||||
|
||||
```python
|
||||
@@ -321,7 +426,7 @@ def answer_question(question:str, text_chunk: str) -> AnswerWithCitation:
|
||||
)
|
||||
```
|
||||
|
||||
## Self Corrections Using Validation Errors
|
||||
## Tying it all together with `instructor.patch()`
|
||||
|
||||
When programming LLMs, having error messages is often desirable. However, with intelligent systems, the ability to correct the output is also crucial. Validators can be valuable in ensuring certain properties of the outputs. The `patch()` method in the `openai` client allows you to use the `max_retries` parameter to specify the number of times you can ask the model to correct the output.
|
||||
|
||||
@@ -377,6 +482,6 @@ In this example, even though there is no code explicitly transforming the name t
|
||||
|
||||
## Conclusion
|
||||
|
||||
In this post, we have explored how validation in AI systems can be simplified by leveraging existing programming concepts. We have demonstrated the use of Pydantic and Instructor to achieve effective validation without introducing new standards or terminology. By utilizing LLM-powered validators and error information, we can prompt adaptive responses and rectify outputs. We encourage you to experiment with validation in your own projects using these powerful tools.
|
||||
We've examined the limitations of traditional validation and how modern tools and AI can offer more robust solutions. From the simplicity of Pydantic and Instructor to the dynamic validation capabilities of LLMs, the landscape of validation is changing but without needing to introduce new contepts. With advanced techniques like validating attributes, chain of thought, and contextual validation, it's clear that the future of validation is not just about preventing bad data but about allowing llms to understand the data and correcting it.
|
||||
|
||||
Remember, validation and error handling are crucial for ensuring the quality and reliability of AI systems. By applying the concepts discussed in this post, you can enhance the control flow and improve the overall performance of your AI applications.
|
||||
Remember, validation and error handling are crucial for ensuring the quality and reliability of AI systems. By applying the concepts discussed in this post, you can enhance the control flow and improve the overall performance of your AI application without introducting new concepts and standards.
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
import instructor
|
||||
import openai
|
||||
from pydantic import BaseModel, Field, model_validator
|
||||
from typing import Optional
|
||||
|
||||
# Enables `response_model` and `max_retries` parameters
|
||||
instructor.patch()
|
||||
|
||||
|
||||
class Validation(BaseModel):
|
||||
is_valid: bool = Field(
|
||||
..., description="Whether the value is valid given the rules"
|
||||
)
|
||||
error_message: Optional[str] = Field(
|
||||
...,
|
||||
description="The error message if the value is not valid, to be used for re-asking the model",
|
||||
)
|
||||
|
||||
|
||||
def validator(values):
|
||||
chain_of_thought = values["chain_of_thought"]
|
||||
answer = values["answer"]
|
||||
resp = openai.ChatCompletion.create(
|
||||
model="gpt-3.5-turbo",
|
||||
messages=[
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are a validator. Determine if the value is valid for the statement. If it is not, explain why.",
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": f"Verify that `{answer}` follows the chain of thought: {chain_of_thought}",
|
||||
},
|
||||
],
|
||||
# this comes from instructor.patch()
|
||||
response_model=Validation,
|
||||
)
|
||||
if not resp.is_valid:
|
||||
raise ValueError(resp.error_message)
|
||||
return values
|
||||
|
||||
|
||||
class Response(BaseModel):
|
||||
chain_of_thought: str
|
||||
answer: str
|
||||
|
||||
@model_validator(mode="before")
|
||||
@classmethod
|
||||
def chain_of_thought_makes_sense(cls, data):
|
||||
return validator(data)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
resp = Response(
|
||||
chain_of_thought="1 + 1 = 2", answer="The meaning of life is 42"
|
||||
)
|
||||
print(resp)
|
||||
except Exception as e:
|
||||
print(e)
|
||||
"""
|
||||
1 validation error for Response
|
||||
Value error, The statement 'The meaning of life is 42' does not follow the chain of thought: 1 + 1 = 2.
|
||||
[type=value_error, input_value={'chain_of_thought': '1 +... meaning of life is 42'}, input_type=dict]
|
||||
"""
|
||||
@@ -0,0 +1,31 @@
|
||||
from pydantic import BaseModel, ValidationError, field_validator, ValidationInfo
|
||||
|
||||
|
||||
class AnswerWithCitation(BaseModel):
|
||||
answer: str
|
||||
citation: str
|
||||
|
||||
@field_validator("citation")
|
||||
@classmethod
|
||||
def remove_stopwords(cls, v: str, info: ValidationInfo):
|
||||
context = info.context
|
||||
if context:
|
||||
text_chunks = context.get("text_chunk")
|
||||
if v not in text_chunks:
|
||||
raise ValueError(f"Citation `{v}` not found in text chunks")
|
||||
return v
|
||||
|
||||
|
||||
try:
|
||||
AnswerWithCitation.model_validate(
|
||||
{"answer": "Jason is a cool guy", "citation": "Jason is cool"},
|
||||
context={"text_chunk": "Jason is just a guy"},
|
||||
)
|
||||
except ValidationError as e:
|
||||
print(e)
|
||||
"""
|
||||
1 validation error for AnswerWithCitation
|
||||
citation
|
||||
Value error, Citation `Jason is cool`` not found in text chunks [type=value_error, input_value='Jason is cool', input_type=str]
|
||||
For further information visit https://errors.pydantic.dev/2.4/v/value_error
|
||||
"""
|
||||
Reference in New Issue
Block a user