ARC-AGI Benchmarking

ARC-AGI Benchmarking is an open-source Python project from the arcprize organization for testing models on ARC-AGI tasks. It runs tasks against model adapters such as OpenAI, Anthropic, Gemini, Fireworks, Grok, OpenRouter and custom providers, with built-in rate limiting, retries, and scoring. It is released under the MIT License.

What It Is

The project is a command-line benchmarking harness. Users point it at a folder of ARC task JSON files, choose a model config, and it writes submissions that can then be scored against the source tasks to produce aggregated results. A local random-baseline adapter allows a dry run without API keys.

How a Run Works

A single-task command (main.py) supports debugging, while cli/run_all.py runs a batch for one model config using asyncio, request pacing and tenacity retries. cli/run_configs.py launches several configs concurrently across datasets such as v1 and v2 public and semi-private evaluation sets, with provider rate limits shared across child processes and an optional cap on in-flight tasks. Settings include number of attempts, retry attempts, and a maximum number of tasks per run. Outputs land in a structured submissions directory, and the scoring script writes results to a results directory.

Models, Config and Logging

Models are defined in models.yml with a config name, provider model id, provider, API key environment variable, optional parameters such as max output tokens and temperature, and per-million-token pricing metadata. Provider rate limits live in provider_config.yml. Raw API request and response events are appended to per-task JSONL logs, which the README notes can contain task data and model reasoning and should be treated as sensitive. New adapters can be added by implementing a ProviderAdapter and registering it.