Cortad

Cortad is an AI research and product company's tool for measuring and improving how AI apps behave. It runs simulated users against your app, asks behavioral questions of every reply, and returns findings tied to conversations and code for a coding agent to act on. An open-source CLI engine (Apache-2.0) is published on GitHub, alongside a hosted product that adds production conversation analysis.

What It Is

Cortad is an evaluation and behavior-monitoring tool for AI applications, which it describes as "like CodeRabbit, but for AI behavior." Simulated users hold realistic conversations with your actual app, and each reply is checked against behavioral questions. The result shows where it failed, how often, the reply that demonstrates the failure, and the file to change.

Cortad does not edit code itself. Coding agents such as Claude Code, Codex or Cursor call Cortad after changing the AI, receive the findings, make the fix, and Cortad reruns to measure the difference.

How the Open-Source CLI Works

The CLI is run with npx: init writes a config file, plan writes simulation files, and a plain run executes them and reports what a change did. Simulations are JSON files describing a persona, the messages they send, and questions with expected answers. They can be edited, extended, and committed to the repository.

Questions are answered in order by code checks, by the app's own trace (for example tool calls), by custom functions from your repository, or by a model that must quote the reply sentence it relied on. Answers are yes, no, could not tell, or did not come up, and the last two are kept out of the failure count.

After a change, the CLI can run only files affected by edits, verify only previously failed cases, report file coverage, and replay stored runs. Output can be SARIF, JUnit XML or JSON, and a GitHub Action is provided for pull request gating. Writers and judges can be local (such as Ollama) or any OpenAI-compatible provider, and an offline mode sends nothing out.

Production Intelligence

The hosted product reads production conversations that reach Cortad through an installed SDK. It tracks metrics such as rules followed and goals achieved, compares user groups and releases, learns user profiles, alerts on regressions, and connects investigations to the prompt or code change involved. Custom metrics can be defined in plain words. The README says this production side is closed source.

Data Handling

Per the README, with the default writer and judge the config and the conversations being judged are sent to Cortad's service, while source code and environment are not. The FAQ states Cortad does not fine-tune models on repositories or conversations.