Attest Framework
Testing framework for AI agents in Python and TypeScript that captures traces, replays recorded model responses, and gates merges in CI.
At a Glance
Free to use under Apache 2.0. Model inference, embedding services, and LLM judges may incur charges from your chosen provider.
Engagement
Available On
Listed Nov 2026
About Attest Framework
Attest is a testing framework for AI agents with SDKs for Python (pytest) and TypeScript (Vitest). It checks an agent's replies, sequences of tool calls, and changes to external state, starting with deterministic checks and adding a model judge only where those cannot express a criterion. The documentation lists version 1.1.0.
What It Is
Attest lets developers write tests for agent behavior inside their existing unit-test runner. Assertions cover the output text, the tools called (which ones, in what order, with which arguments, and how often per business key), and the final world state. Checks report pass, fail or not_evaluated with a reason.
Record, Replay, Gate
The documented workflow has three steps. attest capture records the trace and model exchanges, applies redaction and drafts contracts from the run. Strict replay then runs the agent's real code against recorded model responses through a local proxy, with no live model calls for matching requests. Finally, a CI policy gates the merge: failed tests block by default, and not_evaluated results warn unless the policy sets them to block. The docs note that replay removes live-model variability but not other sources of flaky tests, and that the gate covers only the contracts and baselines you supply.
Six Evaluation Layers
The layers run from free to LLM-judged: schema validation, constraint checks (cost, tokens, latency, steps), trace and behavioral checks, content pattern matching, embedding similarity, and LLM-as-judge. Embedding similarity and judging depend on the chosen provider, with OpenAI, Anthropic and Gemini judges paid and local options available. Plugins add custom assertions that run in the SDK process.
Adapters and Maturity Tiers
Adapters are labeled stable, beta or experimental. OpenAI, Anthropic and Manual adapters are stable in both languages. Gemini, Ollama, LangChain and OpenTelemetry are beta in both, CrewAI is beta in Python, and Google ADK and LlamaIndex are experimental in Python. Multi-agent testing and simulation mode are beta, and continuous evaluation is experimental. The site also includes a migration guide from DeepEval and a refund-desk demo repository.
Community Discussions
Be the first to start a conversation about Attest Framework
Share your experience with Attest Framework, ask questions, or help others learn from your insights.
Pricing
Open Source
Free to use under Apache 2.0. Model inference, embedding services, and LLM judges may incur charges from your chosen provider.
- Python and TypeScript SDKs
- pytest and Vitest integration
- Deterministic assertions and optional model-based evaluation
- Local execution and CI integration
Capabilities
Key Features
- Capture of traces and model exchanges with redaction and drafted contracts
- Strict replay against recorded model responses via a local proxy
- Behavior contracts such as tool cardinality per business key
- Six evaluation layers from schema validation to LLM-as-judge
- CI merge gate with policies and baselines
- Framework and provider adapters for Python and TypeScript
- Multi-agent testing (beta)
- Simulation mode with mock tools, seeded faults and personas (beta)
- Continuous evaluation with Slack or webhook alerts (experimental)
- Custom assertion plugins
- pytest and Vitest integration
