EveryDev.ai
Subscribe
Home
Tools

5,901+ AI tools

  • Featured
  • Gems
  • Trending
  • New
  • Rate tools
  • Compare
  • Arena
Categories
  • Agents2972
  • Coding2248
  • Infrastructure1067
  • Projects818
  • MCP794
  • Research744
  • Analytics565
  • Marketing527
  • Design488
  • Testing476
  • Data442
  • Security412
  • Integration288
  • Communication284
  • Prompts283
  • Extensions238
  • Voice213
  • DevOps190
  • Commerce183
  • Web128
  • Learning123
  • Finance44
AI Tools by Topic
  • AI Coding Assistants
  • Agent Frameworks
  • MCP Servers
  • AI Prompt Tools
  • Vibe Coding Tools
  • AI Design Tools
  • AI Database Tools
  • AI Website Builders
  • AI Testing Tools
  • LLM Evaluations
Follow Us
  • X / Twitter
  • LinkedIn
  • Reddit
  • Discord
  • Threads
  • Bluesky
  • Mastodon
  • YouTube
  • GitHub
  • Instagram
Get Started
  • Users
  • Rate Tools
  • About
  • Editorial Standards
  • Corrections & Disclosures
  • Community Guidelines
  • Advertise
  • Contact Us
  • Newsletter
  • Submit a Tool
  • Start a Discussion
  • Write A Blog
  • Share A Build
  • Terms of Service
  • Privacy Policy
Explore with AI
  • ChatGPT
  • Gemini
  • Claude
  • Grok
  • Perplexity
Agent Experience
  • llms.txt
Theme
With AI, Everyone is a Dev. EveryDev.ai ©
    1. Home
    2. Tools
    3. Attest Framework
    Attest Framework icon

    Attest Framework

    Automated Testing

    Testing framework for AI agents in Python and TypeScript that captures traces, replays recorded model responses, and gates merges in CI.

    Visit Website

    At a Glance

    Pricing
    Open Source

    Free to use under Apache 2.0. Model inference, embedding services, and LLM judges may incur charges from your chosen provider.

    Engagement

    Available On

    API
    SDK
    CLI

    Resources

    WebsiteDocsGitHubllms.txt

    Topics

    Automated TestingLLM EvaluationsAI Development Libraries

    Alternatives

    EvalPlusBigCode Evaluation HarnessOpenAI Evals
    Developer
    attest-framework

    Listed Nov 2026

    About Attest Framework

    Attest is a testing framework for AI agents with SDKs for Python (pytest) and TypeScript (Vitest). It checks an agent's replies, sequences of tool calls, and changes to external state, starting with deterministic checks and adding a model judge only where those cannot express a criterion. The documentation lists version 1.1.0.

    What It Is

    Attest lets developers write tests for agent behavior inside their existing unit-test runner. Assertions cover the output text, the tools called (which ones, in what order, with which arguments, and how often per business key), and the final world state. Checks report pass, fail or not_evaluated with a reason.

    Record, Replay, Gate

    The documented workflow has three steps. attest capture records the trace and model exchanges, applies redaction and drafts contracts from the run. Strict replay then runs the agent's real code against recorded model responses through a local proxy, with no live model calls for matching requests. Finally, a CI policy gates the merge: failed tests block by default, and not_evaluated results warn unless the policy sets them to block. The docs note that replay removes live-model variability but not other sources of flaky tests, and that the gate covers only the contracts and baselines you supply.

    Six Evaluation Layers

    The layers run from free to LLM-judged: schema validation, constraint checks (cost, tokens, latency, steps), trace and behavioral checks, content pattern matching, embedding similarity, and LLM-as-judge. Embedding similarity and judging depend on the chosen provider, with OpenAI, Anthropic and Gemini judges paid and local options available. Plugins add custom assertions that run in the SDK process.

    Adapters and Maturity Tiers

    Adapters are labeled stable, beta or experimental. OpenAI, Anthropic and Manual adapters are stable in both languages. Gemini, Ollama, LangChain and OpenTelemetry are beta in both, CrewAI is beta in Python, and Google ADK and LlamaIndex are experimental in Python. Multi-agent testing and simulation mode are beta, and continuous evaluation is experimental. The site also includes a migration guide from DeepEval and a refund-desk demo repository.

    Attest Framework - 1

    Community Discussions

    Be the first to start a conversation about Attest Framework

    Share your experience with Attest Framework, ask questions, or help others learn from your insights.

    Pricing

    OPEN SOURCE

    Open Source

    Free to use under Apache 2.0. Model inference, embedding services, and LLM judges may incur charges from your chosen provider.

    • Python and TypeScript SDKs
    • pytest and Vitest integration
    • Deterministic assertions and optional model-based evaluation
    • Local execution and CI integration

    Capabilities

    Key Features

    • Capture of traces and model exchanges with redaction and drafted contracts
    • Strict replay against recorded model responses via a local proxy
    • Behavior contracts such as tool cardinality per business key
    • Six evaluation layers from schema validation to LLM-as-judge
    • CI merge gate with policies and baselines
    • Framework and provider adapters for Python and TypeScript
    • Multi-agent testing (beta)
    • Simulation mode with mock tools, seeded faults and personas (beta)
    • Continuous evaluation with Slack or webhook alerts (experimental)
    • Custom assertion plugins
    • pytest and Vitest integration

    Integrations

    pytest
    Vitest
    OpenAI
    Anthropic
    Gemini
    Ollama
    LangChain
    CrewAI
    OpenTelemetry
    Google ADK
    LlamaIndex
    Slack
    API Available
    View Docs

    Ratings & Reviews

    No ratings yet

    Be the first to rate Attest Framework and help others make informed decisions.

    Rate other tools you’ve used

    Developer

    attest-framework

    Read more about attest-framework
    WebsiteGitHub
    1 tool in directory

    Similar Tools

    EvalPlus icon

    EvalPlus

    Open-source evaluation framework that rigorously tests LLM-generated code for correctness and efficiency using HumanEval+, MBPP+ and EvalPerf.

    BigCode Evaluation Harness icon

    BigCode Evaluation Harness

    A framework for evaluating autoregressive code generation language models on a range of code benchmarks.

    OpenAI Evals icon

    OpenAI Evals

    Open-source framework and registry of benchmarks for evaluating LLMs and systems built on LLMs.

    Browse all tools

    Related Topics

    Automated Testing

    AI-powered platforms that automate end-to-end testing processes with intelligent test case generation, execution, and reporting for faster, more reliable software delivery.

    210 tools

    LLM Evaluations

    Platforms and frameworks for evaluating, testing, and benchmarking LLM systems and AI applications. These tools provide evaluators and evaluation models to score AI outputs, measure hallucinations, assess RAG quality, detect failures, and optimize model performance. Features include automated testing with LLM-as-a-judge metrics, component-level evaluation with tracing, regression testing in CI/CD pipelines, custom evaluator creation, dataset curation, and real-time monitoring of production systems. Teams use these solutions to validate prompt effectiveness, compare models side-by-side, ensure answer correctness and relevance, identify bias and toxicity, prevent PII leakage, and continuously improve AI product quality through experiments, benchmarks, and performance analytics.

    246 tools

    AI Development Libraries

    Programming libraries and frameworks that provide machine learning capabilities, model integration, and AI functionality for developers.

    577 tools
    Browse all topics
    Back to all toolsSuggest an edit
    ratings
    discussions