HarnessEval-W

HarnessEval-W is a benchmark and executable evaluation system from MirroS-Lab for judging visual world models. It applies the harness paradigm from the LLM ecosystem: instead of a fixed rubric, a planner routes each case to specialized evaluation skills and sub-agents, and the result is an auditable reasoning trace rather than a bare score. The project provides evaluation code, a public dataset on Hugging Face, and an interactive leaderboard.

What It Is

HarnessEval-W evaluates videos generated by world models, checking whether physics, causality, and world state evolve correctly. Its benchmark covers 330 evaluation cases across exploratory, intentional, and physical transitions, plus drift, revisit, and offscreen persistence cases. The project page lists 11 specialized evaluation skills and 5,940 scored rollouts, and a leaderboard of 18 world models across prompt image-to-video, camera-pose, and native-action interfaces.

How the Evaluation Works

Given a case (initial world, action, probe intent), a planner routes it to the skills that can legitimately evaluate it and records a reason for each skill it skips. Each skill decomposes the question into measurable sub-questions, answered by dedicated sub-agents against rollout evidence. The parent agent validates the evidence and aggregates it into the case score. Routing depends only on case context, never on the model being evaluated. The full trace of questions, answers, scores, and supporting frames is saved as an auditable case card.

Setup and Outputs

Users create three conda environments (launcher/CLI, metric backends, and a physical-plausibility backend), configure credentials, and run the harnesseval eval command on generated videos with a manifest. A harnesseval verify run command checks that a run has all expected scores. A completed run writes summary.json, leaderboard_latest.json and .csv, and LEADERBOARD.md. A bundled example can be evaluated right after setup.

Reported Results and Status

The project page reports a Spearman correlation of 0.93 with human Bradley-Terry ranking on Intentional Transition, and a 4.9x narrower repeat-evaluation envelope than WBench. A 100-case public set and leaderboard were released in September 2026; a hosted submission and evaluation service is still listed as a TODO. The repository invites contributions of cases, skills, and models.