# ARC-AGI Benchmarking

> Python harness for running ARC-AGI tasks against multiple LLM provider adapters with rate limiting, retries, and scoring.

ARC-AGI Benchmarking is an open-source Python project from the arcprize organization for testing models on ARC-AGI tasks. It runs tasks against model adapters such as OpenAI, Anthropic, Gemini, Fireworks, Grok, OpenRouter and custom providers, with built-in rate limiting, retries, and scoring. It is released under the MIT License.

## What It Is

The project is a command-line benchmarking harness. Users point it at a folder of ARC task JSON files, choose a model config, and it writes submissions that can then be scored against the source tasks to produce aggregated results. A local random-baseline adapter allows a dry run without API keys.

## How a Run Works

A single-task command (main.py) supports debugging, while cli/run_all.py runs a batch for one model config using asyncio, request pacing and tenacity retries. cli/run_configs.py launches several configs concurrently across datasets such as v1 and v2 public and semi-private evaluation sets, with provider rate limits shared across child processes and an optional cap on in-flight tasks. Settings include number of attempts, retry attempts, and a maximum number of tasks per run. Outputs land in a structured submissions directory, and the scoring script writes results to a results directory.

## Models, Config and Logging

Models are defined in models.yml with a config name, provider model id, provider, API key environment variable, optional parameters such as max output tokens and temperature, and per-million-token pricing metadata. Provider rate limits live in provider_config.yml. Raw API request and response events are appended to per-task JSONL logs, which the README notes can contain task data and model reasoning and should be treated as sensitive. New adapters can be added by implementing a ProviderAdapter and registering it.

## Features
- Run ARC-AGI tasks against multiple model provider adapters
- Built-in rate limiting and retries
- Scoring of submissions against task solutions
- Random-baseline adapter for dry runs without API keys
- Batch runs with asyncio and request pacing
- Multi-config launcher with shared provider rate limits and concurrency cap
- Model configs in models.yml with pricing metadata
- Per-task raw API JSONL logging
- Optional metrics collection
- Extensible provider adapters

## Integrations
OpenAI, Anthropic, Gemini, Fireworks, Grok, OpenRouter, X.AI, Hugging Face

## Platforms
API, CLI

## Pricing
Open Source

## Links
- Website: https://github.com/arcprize/arc-agi-benchmarking
- Documentation: https://github.com/arcprize/arc-agi-benchmarking
- Repository: https://github.com/arcprize/arc-agi-benchmarking
- EveryDev.ai: https://www.everydev.ai/tools/arc-agi-benchmarking