Regent

Regent is an LLM regression testing tool built for production teams shipping AI-powered applications. It proxies every LLM call inside your app — including nested chains, multi-step agents, and parallel calls — and automatically diffs outputs between your current branch and the main branch baseline. Results are posted directly as PR comments, so your team knows exactly what changed before merging.

• Automatic baseline capture: Regent captures your main branch as the golden standard on setup and re-captures it automatically on every merge, keeping baselines always up to date.
• Full call chain visibility: Unlike tools that only inspect final outputs, Regent captures every intermediate LLM call in a request — nested chains, multi-step agents, and parallel calls are all tracked.
• PR comments out of the box: Every pull request automatically receives a detailed diff report as a comment, showing exactly which LLM calls drifted and how outputs changed.
• Zero production traffic: Regent only proxies CI test runs — your live production traffic is never touched or routed through Regent.
• Scenario-based testing: Define the API endpoints you want to monitor, and Regent reruns those scenarios on every PR to compare against the baseline.
• Drift detection: Regent highlights specific field-level changes in LLM outputs (e.g., tone, confidence, word count, department routing) so regressions are immediately actionable.

To get started, connect your GitHub repository, add the Regent workflow file, and define the API endpoint scenarios you want to test. Regent handles baseline capture and diff reporting automatically from there.