Reame

Reame is an MIT-licensed LLM inference server built by Marco Caciotti (swellweb) on top of llama.cpp, designed specifically for CPU-first deployment on free tiers, shared VPS instances, and 2-core ARM boxes. It exposes an OpenAI-compatible REST API and ships a zero-config CLI that downloads models, auto-configures threads and KV cache, and starts serving in a single command. The project reached v0.1.5 in July 2026 and is actively developed.

What It Is

Reame is a self-hosted local inference server whose central thesis is: on a CPU, never compute the same thing twice. It targets narrow, repetitive AI workloads over private data — document extraction, batch tagging, RAG pipelines, privacy-bound legal or medical text — where the answer lives in the provided context rather than in broad model knowledge. The README reports 100% accuracy on long-context extraction with a 7B model on a free 2-core ARM box, and positions Reame as a cost-effective alternative to metered API calls for thin-margin SaaS products.

Core Caching Architecture

Reame's performance model rests on several layered caching mechanisms:

• Persistent shared-prefix KV cache — prompts are split into fixed token blocks; a chain hash keys a KV snapshot at every block boundary. Different prompts sharing a prefix restore the longest cached boundary and decode only their own tail. Snapshots are stored on NVMe with zstd compression, LRU budgeting, and checksums, and survive restarts.
• Palimpsest generation archive — every completed generation feeds an on-disk n-gram archive; future requests draft from it at zero inference cost. The README reports a measured 2.3× throughput improvement (22→51 tok/s) on an M3 Pro.
• ARCA shared-memory daemon — a Redis-compatible service (reame arca) that provides an exact-response cache (~0.02 s vs ~1 s of inference) and a fleet-wide generation corpus so one node's output can draft another's.

Speculative Decoding and Quality Features

Beyond caching, Reame includes several inference acceleration and quality mechanisms:

• Self-regulating speculative decoding — a small draft model or zero-cost n-gram lookup proposes tokens; the target verifies them in one batched pass. A feedback controller measures whether speculation pays on the current hardware and disables it automatically when it doesn't.
• Il Suggeritore — inverts grammar-constrained decoding to propose structure tokens (list numbering, bullets, format tokens) for free on novel content.
• The Conclave — --best-of N generates N candidate answers in one interleaved batch using KV cloning (one prefill, cloned into N attempts), then elects a winner by majority vote. The README reports it squeezes roughly one extra correct answer per quiz out of the same model, correcting variance rather than bias.
• Interleaved multi-user serving — N concurrent generations advance together inside single multi-sequence batches, sharing every model-weight read.

Deployment and Setup Path

Installation is available via Homebrew (brew tap swellweb/reame && brew install reame), prebuilt binaries for Linux x64/arm64 and macOS arm64 on the GitHub releases page, and build-from-source with CMake ≥ 3.16 and a C++17 compiler. The CLI resolves catalog names like qwen2.5-1.5b, downloads to ~/.reame/models on first use, and auto-configures threads, KV quantization, and cache directory. A config file is only needed for fine-grained control. The server exposes /v1/completions, /v1/chat/completions, SSE streaming, session snapshots, bearer auth, and a /metrics endpoint, making it a drop-in target for any OpenAI client.

Update: v0.1.5

The latest release is v0.1.5, published on 2026-07-13. The repository was created in early July 2026 and has seen active pushes through mid-July 2026. The project is explicitly described as "young and deliberately opinionated and focused" — CPU-only serving, one model per process, correctness pinned by 220+ isolated test cases. Planned roadmap items mentioned in the README include warm-ahead prefill, a semantic (L2) cache layer for the ARCA daemon, and first-class MoE serving. GPU offload, training, and model management UX are explicitly out of scope.