# Cortad

> Tests AI apps with simulated users and analyzes production conversations so coding agents can fix AI behavior and verify the results.

Cortad is an AI research and product company's tool for measuring and improving how AI apps behave. It runs simulated users against your app, asks behavioral questions of every reply, and returns findings tied to conversations and code for a coding agent to act on. An open-source CLI engine (Apache-2.0) is published on GitHub, alongside a hosted product that adds production conversation analysis.

## What It Is

Cortad is an evaluation and behavior-monitoring tool for AI applications, which it describes as "like CodeRabbit, but for AI behavior." Simulated users hold realistic conversations with your actual app, and each reply is checked against behavioral questions. The result shows where it failed, how often, the reply that demonstrates the failure, and the file to change.

Cortad does not edit code itself. Coding agents such as Claude Code, Codex or Cursor call Cortad after changing the AI, receive the findings, make the fix, and Cortad reruns to measure the difference.

## How the Open-Source CLI Works

The CLI is run with npx: `init` writes a config file, `plan` writes simulation files, and a plain run executes them and reports what a change did. Simulations are JSON files describing a persona, the messages they send, and questions with expected answers. They can be edited, extended, and committed to the repository.

Questions are answered in order by code checks, by the app's own trace (for example tool calls), by custom functions from your repository, or by a model that must quote the reply sentence it relied on. Answers are yes, no, could not tell, or did not come up, and the last two are kept out of the failure count.

After a change, the CLI can run only files affected by edits, verify only previously failed cases, report file coverage, and replay stored runs. Output can be SARIF, JUnit XML or JSON, and a GitHub Action is provided for pull request gating. Writers and judges can be local (such as Ollama) or any OpenAI-compatible provider, and an offline mode sends nothing out.

## Production Intelligence

The hosted product reads production conversations that reach Cortad through an installed SDK. It tracks metrics such as rules followed and goals achieved, compares user groups and releases, learns user profiles, alerts on regressions, and connects investigations to the prompt or code change involved. Custom metrics can be defined in plain words. The README says this production side is closed source.

## Data Handling

Per the README, with the default writer and judge the config and the conversations being judged are sent to Cortad's service, while source code and environment are not. The FAQ states Cortad does not fine-tune models on repositories or conversations.

## Features
- Simulated users that converse with your actual app
- Up to 100 behavioral questions per reply
- Editable JSON simulation files
- Code, trace, custom function and model-based judging
- Verify mode that reruns only failed cases
- Run history with comparison against previous runs
- Changed-file targeted runs and coverage reporting
- Question packs for kinds of apps
- Eval file import
- SARIF, JUnit and JSON output
- GitHub Action for CI
- Bring-your-own writer and judge models including Ollama
- Offline mode
- Production conversation analysis via SDK
- User behavior profiles and release comparison
- Custom metrics defined in plain words

## Integrations
Claude Code, Codex, Cursor, GitHub Actions, Ollama, OpenAI-compatible providers, OpenRouter

## Platforms
CLI, WEB, API

## Pricing
Open Source, Free tier available

## Links
- Website: https://cortad.com
- Documentation: https://cortad.com/docs/quickstart
- Repository: https://github.com/cortad/cortad
- EveryDev.ai: https://www.everydev.ai/tools/cortad