# Scenario

> An open-source agent testing framework that simulates realistic user conversations to test AI agents end-to-end across any framework, with support for Python, TypeScript, and Go.

Scenario is an open-source agent testing framework built by LangWatch that uses simulation-based testing to validate AI agent behavior end-to-end. Released under the Apache 2.0 license, it is available in Python, TypeScript, and Go, and integrates with any agent framework through a single `call()` method adapter interface.

## What It Is

Scenario fills the "Level 3" layer of agent quality assurance — end-to-end simulation testing — sitting above traditional unit tests and LLM evals. Rather than testing individual components in isolation, Scenario runs a simulated user against a real agent across multi-turn conversations, then uses a Judge Agent to evaluate whether the agent met defined success criteria. The framework is designed to be agnostic: it works alongside any LLM eval library and any agent framework, requiring no proprietary dataset to get started.

## How the Simulation Loop Works

A Scenario test defines a natural-language description of a situation, assigns agents (the system under test, a UserSimulatorAgent, and optionally a JudgeAgent), and optionally provides a `script` list that controls the conversation flow. Each script step is an arbitrary function receiving the current `ScenarioState`, so teams can:

- Let the user simulator generate realistic messages automatically
- Hardcode specific user or agent turns for deterministic coverage
- Assert tool calls at precise points in the conversation
- Inject custom evals from external libraries mid-conversation
- Use `scenario.proceed(turns=N)` to run N turns on autopilot
- Trigger `scenario.judge()` to request a verdict at any point

Without a script, the simulator runs fully on autopilot until the judge issues a pass/fail verdict or the maximum turn count is reached.

## Agent Integration and Framework Support

Scenario integrates with agents by wrapping them in an `AgentAdapter` that exposes a single `call()` method. The documentation lists first-class integration guides for: LangGraph, CrewAI, Pydantic AI, Agno, Google ADK, Inngest AgentKit, LiteLLM, Mastra, OpenAI SDK, Vercel AI SDK, and Claude Code. Remote agents can be tested over HTTPS, streaming, Server-Sent Events, and stateful thread-ID protocols. Multimodal inputs (images, files) are also supported.

## Voice Agent Testing

Scenario treats voice as a first-class citizen using the same `scenario.run()` entrypoint and script DSL. Shipped platform adapters include ElevenLabs (hosted Conversational AI), OpenAI Realtime, Twilio Media Streams, Pipecat WebSocket, and Gemini Live. The voice surface adds `scenario.audio()` for injecting recorded clips, `scenario.background_noise()` for noise injection, `scenario.interrupt()` for interruption testing, and `result.latency` with TTFB, p50, and p95 metrics.

## Red Teaming

Scenario ships a `RedTeamAgent` as a drop-in replacement for the user simulator. It runs multi-turn adversarial attacks using the Crescendo escalation strategy, with per-turn scoring, refusal detection, and backtracking, all within the same `scenario.run()` loop and CI pipeline. This makes it straightforward to surface safety and security vulnerabilities in agents without a separate toolchain.

## Update: python/v1.0.0

The repository reached its v1.0.0 milestone for the Python package on July 24, 2026. The project was created in April 2025 and has been actively developed, with the last push recorded on August 3, 2026. The GitHub repository had accumulated 939 stars and 71 forks at the time of indexing. The open-source SDK (Python + TypeScript) is available on PyPI as `langwatch-scenario` and on npm as `@langwatch/scenario`. LangWatch also offers a cloud visualization platform that connects to Scenario via an API key for real-time simulation replay and team collaboration, with its own separate pricing tiers.

## Features
- Simulation-based agent testing
- LLM-powered user simulator
- Judge agent with configurable success criteria
- Multi-turn conversation testing
- Scripted and autopilot simulation modes
- Tool-call verification across long dialogues
- Framework-agnostic adapter (one call() method)
- Python, TypeScript, and Go support
- CI/CD integration (pytest, vitest, jest)
- Voice agent testing (ElevenLabs, OpenAI Realtime, Twilio, Pipecat, Gemini Live)
- Red teaming with Crescendo escalation strategy
- Debug mode with step-through and intervention
- Deterministic caching for reproducible tests
- Async-native parallelism
- Simulation visualizer for real-time debugging
- Multimodal support (images, files)
- Remote agent testing (HTTPS, streaming, SSE, stateful)
- Custom judge and custom observability
- Latency metrics (TTFB, p50, p95) for voice agents

## Integrations
LangGraph, CrewAI, Pydantic AI, Agno, Google ADK, Inngest AgentKit, LiteLLM, Mastra, OpenAI SDK, Vercel AI SDK, Claude Code, ElevenLabs, OpenAI Realtime, Twilio Media Streams, Pipecat, Gemini Live, pytest, Vitest, Jest, LangWatch Cloud, GitHub Actions

## Platforms
WINDOWS, WEB, API, DEVELOPER_SDK, CLI

## Pricing
Open Source, Free tier available

## Version
python/v1.0.0

## Links
- Website: https://scenario.langwatch.ai
- Documentation: https://scenario.langwatch.ai
- Repository: https://github.com/langwatch/scenario
- EveryDev.ai: https://www.everydev.ai/tools/scenario-langwatch