LMCache

LMCache is an open-source KV cache management layer for LLM inference, developed as a community project under the LMCache GitHub organization. It persists KV cache beyond GPU memory and reuses it across requests, sessions and serving engines, which the project says reduces time-to-first-token and improves throughput for long-context, multi-turn and RAG workloads. The latest release listed is v0.5.5, and the code is licensed under Apache 2.0.

What It Is

LMCache sits between an inference engine and storage, treating KV cache as reusable data rather than temporary state. Instead of recomputing prefill for repeated prompts, conversation history or retrieved content, the engine can load previously computed KV cache from CPU memory, local disk or external backends. The project describes itself as vendor-neutral, so users can switch serving engines and storage vendors while reusing stored caches. It installs with pip install lmcache.

Deployment Modes

LMCache supports an in-process mode, where it runs inside the inference engine process, and a multiprocess mode, where it runs as a standalone server separate from the engine. The multiprocess mode keeps cache available if an engine worker crashes or restarts. The project site names it the recommended path and the focus of future development.

Caching Capabilities

• Tiered offloading across CPU RAM, local storage and remote backends such as Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL and GDS.
• Non-prefix KV reuse through CacheBlend, which selectively recomputes tokens to recover quality.
• KV transfer from prefill to decode workers over NVLink, RDMA or TCP for prefill/decode disaggregation.
• A pluggable SERDE interface for compression, token dropping and custom serialization.
• Observability metrics covering health, prefix cache hits, cache lifecycle and per-user usage.

Research Background

The project site says LMCache grew out of systems research at the University of Chicago, and it links the LMCache, CacheGen and CacheBlend papers. The README says its development is supported in part by Tensormesh.