OpenAI Evals

OpenAI Evals is a framework from OpenAI for evaluating large language models and systems built with them. It includes a registry of existing evals that test different dimensions of OpenAI models, and lets users write custom or private evals for their own workflows. The README also notes that evals can now be configured and run in the OpenAI Dashboard.

What It Is

Evals is a Python framework (Python 3.9 or later) for running and building LLM evaluations. Users install it via pip, supply an OpenAI API key, and run evals from the registry locally. Private evals can be built from a user's own data without exposing that data publicly.

How Evals Are Built and Run

Basic and model-graded evals can be created from templates without writing evaluation code: data is provided in JSON and parameters are specified in YAML. For more advanced cases such as prompt chains or tool-using agents, the Completion Function Protocol is available. Custom eval logic can also be implemented in Python. The registry data is stored with Git-LFS, and users can fetch all of it or only selected evals.

Results and Contribution

Eval results can optionally be logged to a Snowflake database, and evals can also be run and created using Weights & Biases. The repository accepts contributed evals through pull requests, though it states it is currently not accepting evals with custom code; model-graded evals with custom YAML files are still accepted. Contributions fall under the repository's MIT license terms.