instructor/docs/concepts/patching.md

Patching

Instructor enhances client functionality with three new keywords for backwards compatibility. This allows use of the enhanced client as usual, with structured output benefits.

• response_model: Defines the response type for chat.completions.create.
• max_retries: Determines retry attempts for failed chat.completions.create validations.
• validation_context: Provides extra context to the validation process.

There are three methods for structured output:

1. Function Calling: The primary method. Use this for stability and testing.
2. Tool Calling: Useful in specific scenarios; lacks the reasking feature of OpenAI's tool calling API.
3. JSON Mode: Offers closer adherence to JSON but with more potential validation errors. Suitable for specific non-function calling clients.

Function Calling

from openai import OpenAI
import instructor

client = instructor.patch(OpenAI())

Tool Calling

import instructor
from instructor import Mode

client = instructor.patch(OpenAI(), mode=Mode.TOOLS)

JSON Mode

import instructor
from instructor import Mode
from openai import OpenAI

client = instructor.patch(OpenAI(), mode=Mode.JSON)

Markdown JSON Mode

!!! warning "Experimental"

This is not recommended, and may not be supported in the future, this is just left to support vision models.

import instructor
from instructor import Mode
from openai import OpenAI

client = instructor.patch(OpenAI(), mode=Mode.MD_JSON)

Schema Integration

In JSON Mode, the schema is part of the system message:

import instructor
from openai import OpenAI

client = instructor.patch(OpenAI())

response = client.chat.completions.create(
    model="gpt-3.5-turbo-1106",
    response_format={"type": "json_object"},
    messages=[
        {
            "role": "system",
            "content": f"Match your response to this json_schema: \n{UserExtract.model_json_schema()['properties']}",
        },
        {
            "role": "user",
            "content": "Extract jason is 25 years old",
        },
    ],
)
user = UserExtract.from_response(response, mode=Mode.JSON)
assert user.name.lower() == "jason"
assert user.age == 25

Understanding the Chat Completion Parametsrs

Mode: FUNCTIONS

• Adds functions and function_call keys with OpenAI schema information.
• No direct content change in messages, but function-based response processing is guided.

if mode == Mode.FUNCTIONS:
    chat_completion_parameters["functions"] = [response_model.openai_schema]
    chat_completion_parameters["function_call"] = {"name": response_model.openai_schema["name"]}

Mode: TOOLS

• Adds tools and tool_choice keys with tool details following the OpenAI schema.
• Messages aren't modified in content; tool-based response processing is guided.

if mode == Mode.TOOLS:
    chat_completion_parameters["tools"] = [{"type": "function", "function": response_model.openai_schema}]
    chat_completion_parameters["tool_choice"] = {"type": "function", "function": {"name": response_model.openai_schema["name"]}}

Mode: JSON

• Appends response_format to indicate a JSON object type.
• Adds or modifies a system message for JSON format adherence and schema instructions.

if mode == Mode.JSON:
    chat_completion_parameters["response_format"] = {"type": "json_object"}
    chat_completion_parameters["messages"].append({"role": "system", "content": "Provide response in JSON format adhering to the specified schema."})

Mode: MD_JSON

• Similar to JSON mode, but with Markdown code block formatting.
• Adds a stop sequence for the Markdown JSON response and system message for instructions.

if mode == Mode.MD_JSON:
    chat_completion_parameters["response_format"] = {"type": "json_object"}
    chat_completion_parameters["stop"] = "```"
    chat_completion_parameters["messages"].append({"role": "system", "content": "Provide response in Markdown-formatted JSON within the code block."})

Mode: JSON_SCHEMA

• Appends response_format with a JSON object type and specific schema.
• Modifies system messages for JSON schema formatting instructions.
• Only supported by Anyscale!

if mode == Mode.JSON_SCHEMA:
    chat_completion_parameters["response_format"] = {"type": "json_object", "schema": response_model.model_json_schema()}
    chat_completion_parameters["messages"].append({"role": "system", "content": "Format the response according to the following JSON schema: " + str(response_model.model_json_schema())})

In each mode, the chat completion parameters are adapted to ensure the assistant's response adheres to the specific format required.