kennethreitz/instructor
1
0
Fork 0
mirror of https://github.com/kennethreitz/instructor.git synced 2026-06-05 22:50:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

add evals

Browse Source
This commit is contained in:
Jason Liu
2023-11-25 20:02:15 -05:00
parent d3a567f2ff
commit 797f623180
2 changed files with 13 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/contributing.md
+4
View File
@@ -1,5 +1,9 @@
We would love for you to contribute to `Instructor`.
## [Evals](https://github.com/jxnl/instructor/tree/main/tests/openai/evals)
We invite you to contribute evals in pytest as a way to monitor the quality of the openai models and the instructor library. To get started check out the [jxnl/instructor/tests/evals](https://github.com/jxnl/instructor/tree/main/tests/openai/evals) and contribute your own evals in the form of pytest tests. These evals will be run once a week and the results will be posted.
## Issues
If you find a bug, please file an issue on [our issue tracker on GitHub](https://github.com/jxnl/instructor/issues).
tests/openai/evals/readme.md
+9
View File
@@ -0,0 +1,9 @@
# How to Contribute: Writing and Running Evaluation Tests
We welcome contributors to expand our suite of evaluation tests for data extraction. This guide provides instructions on creating tests with `pytest`, `pydantic`, and other tools, focusing on broad coverage and failure modalities understanding.
## Define Test Scenarios
Identify data extraction scenarios relevant to you. Create test cases with inputs and expected outputs.
Reference the `test_extract_users.py` which contains a test case for extracting users, using all models and all modes. The test case is parameterized with the model and mode, and the test function is parameterized with the input and expected output.