Consumer Agent Testing Tool (CATT)

Consumer Agent Testing Tool (CATT) is an open-source Python tool from Coverage Cat, Inc. that tests how an assistant uses a service. It runs a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API, then checks what the service actually did and records the evidence. The latest release listed on GitHub is v0.4.0.

What It Is

CATT is a surrogate test host: it approximates an assistant's host environment, not its private implementation. Muse and Instinct are both supported as explicit emulation targets, and neither is the default. The README states that a passing test does not certify compatibility with Muse or any hosted assistant, and that the project is not affiliated with Meta, Instinct, OpenAI or Anthropic.

How a Test Runs

Authors write a scenario, optionally scaffolded from a live MCP tool catalog with the scaffold command, defining the user goal, allowed tools, approval rules, host-private values and outcome checks. connector-run replays scripted calls with no model, or lets a model-driven agent choose steps and run multiple trials to report pass^k. Checks read the service's real outcome, through a private state endpoint or a read tool, and verify that sent calls matched approval cards and that the final answer made no false claims. Other lanes cover embedded MCP apps rendered in Chromium (surface-run), restricted browser runs (browse) and mapped HTTP APIs.

Readiness Checks and Evidence

The check command reviews catalogs and documents without running an agent, including Muse connector guidelines, a shared surrogate contract, Personal Agent Protocol manifests, gadget registration messages and MCP protocol contracts. Runs are sealed, and report, export, replay and diff let teams inspect results, share redacted exports and compare builds for outcome regressions. Scripted runs can start the target service, write JUnit and JSON summaries, and run in CI.

Setup Path

CATT needs Python 3.12 or newer, uv, and Node.js 20 or newer; Chromium is installed for the browser and MCP Apps lanes. The README says the recorded release evidence covers macOS, with Linux commands provided but not qualified. Agent runs need a model profile, with OpenRouter and Meta Model API adapters documented.