inferock-bench

inferock-bench is an open-source local proxy built by OpiusAI that sits between your application and AI provider APIs, recording independent per-call receipts with spend, bill-bounded money loss, time loss, and invoice-check exposure. It supports OpenAI, Anthropic (Claude), Gemini Developer API, and pinned OpenRouter endpoints, and is distributed as an npm package runnable via npx inferock-bench. The project launched in July 2026 and has accumulated 127 GitHub stars and 49 forks as of mid-August 2026.

What It Is

inferock-bench is a local diagnostic measurement proxy for metered LLM API traffic. It intercepts provider calls routed through localhost, attaches your provider API key only to outbound provider requests, and writes local event records that are graded by the @inferock/measure library against The Inferock Standard — a published, versioned rulebook for what counts as a billable failure, what is provider-recognized recoverable, and what stays as invoice-check exposure. The tool addresses a specific gap: AI providers report usage totals but do not give customers per-call receipts that cross-check delivery evidence against charges.

How the Receipt Works

Every proxied call produces a structured receipt with four labeled headline numbers:

• spent — provider spend observed by the run for priced calls the proxy saw
• money loss — bill-bounded dollar loss tied to observed spend or charge evidence under The Inferock Standard
• time loss — real wait or downtime measured as time, never added to dollars
• invoice-check exposure — amounts like cache-discount-at-risk, labeled "verify your invoice" and never summed into money loss

The receipt also reports provider-recognized recovery, a bill-bounded recognition gap, and a coverage line such as surfaces watched 10/12 | signals 3 | not-openable 2. A zero only counts for a watched surface; unopened surfaces are named as coverage debt rather than silently claimed clean.

Supported Providers and Integrations

The tool measures four provider planes: OpenAI, Anthropic, Gemini Developer API, and pinned OpenRouter endpoints covering meta-llama, deepseek, mistral, moonshot/kimi, z-ai/glm, and qwen on observed hosts. Integration guides exist for Claude Code, the OpenAI SDK, the Gemini Developer API, OpenRouter, and CI/headless usage. Pointing an existing SDK at the local proxy requires changing only two settings — apiKey (to the local ibl_ bench key) and baseURL (to http://127.0.0.1:4318).

Architecture and Key Boundary

inferock-bench runs as a local Node.js process with a browser dashboard. Provider keys are saved locally under ~/.inferock-bench/ with owner-only file permissions, are shown back only in masked form, and are attached only to outbound provider requests — they are not sent to Inferock's hosted service. The generated local ibl_ bench key is a local-only credential. The @inferock/measure grading library is Apache-2.0 licensed and ships separately on npm; the benchmark CLI uses FSL-1.1-ALv2 (converting to Apache-2.0 after two years); The Inferock Standard documents are CC-BY-4.0.

Update: Public Run Card 2026-08-05

The current cumulative public ledger (as of the 2026-08-05 addendum) contains 1,303 measured calls across OpenAI, Anthropic, Gemini, and pinned OpenRouter coverage, with $8.43 provider spend observed, $0.03 bill-bounded money loss (0.3%), approximately 2.9 minutes of time loss, and $18.88 cache-discount-at-risk invoice-check exposure kept separate from money loss. The current receipt watches 12 of 13 surfaces. The public receipt presentation was introduced in version 0.1.10 and re-rendered in the 0.2.4 Nominal Light product UI. The project also includes a pre-launch Reliability Index feature that lets users opt in locally to preview an anonymized aggregate payload before any data is sent to a public backend.

Why It Matters

The README cites a third-party audit firm report from June 2026 claiming it reviewed approximately $34M of AI invoices and found approximately $1.7M in overbilling — the source attribution and caveats are linked in the spec annex. The tool's stated purpose is to give customers independent, per-call evidence rather than relying solely on provider-reported totals. Providers including Anthropic and OpenAI have issued denials of widespread overbilling; the README includes these as scope boundaries and treats them as claims to test against local per-call evidence rather than as admissions.