Scenario

Scenario is an open-source agent testing framework built by LangWatch that uses simulation-based testing to validate AI agent behavior end-to-end. Released under the Apache 2.0 license, it is available in Python, TypeScript, and Go, and integrates with any agent framework through a single call() method adapter interface.

What It Is

Scenario fills the "Level 3" layer of agent quality assurance — end-to-end simulation testing — sitting above traditional unit tests and LLM evals. Rather than testing individual components in isolation, Scenario runs a simulated user against a real agent across multi-turn conversations, then uses a Judge Agent to evaluate whether the agent met defined success criteria. The framework is designed to be agnostic: it works alongside any LLM eval library and any agent framework, requiring no proprietary dataset to get started.

How the Simulation Loop Works

A Scenario test defines a natural-language description of a situation, assigns agents (the system under test, a UserSimulatorAgent, and optionally a JudgeAgent), and optionally provides a script list that controls the conversation flow. Each script step is an arbitrary function receiving the current ScenarioState, so teams can:

• Let the user simulator generate realistic messages automatically
• Hardcode specific user or agent turns for deterministic coverage
• Assert tool calls at precise points in the conversation
• Inject custom evals from external libraries mid-conversation
• Use scenario.proceed(turns=N) to run N turns on autopilot
• Trigger scenario.judge() to request a verdict at any point

Without a script, the simulator runs fully on autopilot until the judge issues a pass/fail verdict or the maximum turn count is reached.

Agent Integration and Framework Support

Scenario integrates with agents by wrapping them in an AgentAdapter that exposes a single call() method. The documentation lists first-class integration guides for: LangGraph, CrewAI, Pydantic AI, Agno, Google ADK, Inngest AgentKit, LiteLLM, Mastra, OpenAI SDK, Vercel AI SDK, and Claude Code. Remote agents can be tested over HTTPS, streaming, Server-Sent Events, and stateful thread-ID protocols. Multimodal inputs (images, files) are also supported.

Voice Agent Testing

Scenario treats voice as a first-class citizen using the same scenario.run() entrypoint and script DSL. Shipped platform adapters include ElevenLabs (hosted Conversational AI), OpenAI Realtime, Twilio Media Streams, Pipecat WebSocket, and Gemini Live. The voice surface adds scenario.audio() for injecting recorded clips, scenario.background_noise() for noise injection, scenario.interrupt() for interruption testing, and result.latency with TTFB, p50, and p95 metrics.

Red Teaming

Scenario ships a RedTeamAgent as a drop-in replacement for the user simulator. It runs multi-turn adversarial attacks using the Crescendo escalation strategy, with per-turn scoring, refusal detection, and backtracking, all within the same scenario.run() loop and CI pipeline. This makes it straightforward to surface safety and security vulnerabilities in agents without a separate toolchain.

Update: python/v1.0.0

The repository reached its v1.0.0 milestone for the Python package on July 24, 2026. The project was created in April 2025 and has been actively developed, with the last push recorded on August 3, 2026. The GitHub repository had accumulated 939 stars and 71 forks at the time of indexing. The open-source SDK (Python + TypeScript) is available on PyPI as langwatch-scenario and on npm as @langwatch/scenario. LangWatch also offers a cloud visualization platform that connects to Scenario via an API key for real-time simulation replay and team collaboration, with its own separate pricing tiers.