Inspect Petri

Inspect Petri is an alignment auditing agent from Meridian Labs, built on Inspect AI, that automates interaction with language models to detect potential alignment issues, reward hacking, and other concerning behaviors. The documentation describes it as a collaboration between Meridian Labs and the UK AISI Red Team, based on work originally done by the Alignment team at Anthropic. It is released under the MIT license; the latest release listed is 3.1.1.

What It Is

Petri is a framework for testing concrete alignment hypotheses end-to-end. A user supplies seed instructions, and Petri generates realistic audit scenarios from them. It runs multi-turn conversations between an auditor model and a target model, simulates tools, supports rollbacks, and then has a judge model score the transcript against a consistent rubric.

How an Audit Works

Three model roles are involved: an auditor that designs and drives the audit, a target that is being evaluated, and a judge that scores the result. Audits are run with the inspect eval inspect_petri/audit command, where models are assigned to each role with --model-role flags, or from Python using the audit function. Options include max turns (default 30), prefill, rollback, skills, a realism filter, and judge dimensions.

Seeds and Judging Dimensions

Petri ships with over 170 built-in conversation seeds, which can be used in full, filtered by tag or id, or replaced with custom seeds written as markdown files. The judge scores transcripts on 38 built-in dimensions, such as unprompted sycophancy, concerning behavior, admirable behavior, and eval awareness. Each dimension receives a 1-10 score with a written justification citing specific messages. Custom dimensions can also be defined.

Viewing Results and Customization

Audit logs are written to a local ./logs directory and viewed with inspect view, which shows the sample list with scores, a target view of the conversation, and an auditor view showing the auditor's reasoning and tool calls. The auditor can be extended with skills (directories containing a SKILL.md file), custom approvals, and custom auditor tools, and advanced workflows can be built with custom tasks. Extensions include Petri Bloom, which generates evaluation suites for specific behaviors, and Petri Dish.

Responsible Use

The docs note that Petri tests for concerning behaviors that can involve harmful content, and model providers may block accounts that generate too many harmful requests. A full default run of 170+ seeds is described as typically taking a few hours with non-trivial API spend.