# LMCache

> Open-source KV cache management layer that stores, reuses, compresses and moves KV cache to speed up LLM inference.

LMCache is an open-source KV cache management layer for LLM inference, developed as a community project under the LMCache GitHub organization. It persists KV cache beyond GPU memory and reuses it across requests, sessions and serving engines, which the project says reduces time-to-first-token and improves throughput for long-context, multi-turn and RAG workloads. The latest release listed is v0.5.5, and the code is licensed under Apache 2.0.

## What It Is

LMCache sits between an inference engine and storage, treating KV cache as reusable data rather than temporary state. Instead of recomputing prefill for repeated prompts, conversation history or retrieved content, the engine can load previously computed KV cache from CPU memory, local disk or external backends. The project describes itself as vendor-neutral, so users can switch serving engines and storage vendors while reusing stored caches. It installs with pip install lmcache.

## Deployment Modes

LMCache supports an in-process mode, where it runs inside the inference engine process, and a multiprocess mode, where it runs as a standalone server separate from the engine. The multiprocess mode keeps cache available if an engine worker crashes or restarts. The project site names it the recommended path and the focus of future development.

## Caching Capabilities

- Tiered offloading across CPU RAM, local storage and remote backends such as Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL and GDS.
- Non-prefix KV reuse through CacheBlend, which selectively recomputes tokens to recover quality.
- KV transfer from prefill to decode workers over NVLink, RDMA or TCP for prefill/decode disaggregation.
- A pluggable SERDE interface for compression, token dropping and custom serialization.
- Observability metrics covering health, prefix cache hits, cache lifecycle and per-user usage.

## Research Background

The project site says LMCache grew out of systems research at the University of Chicago, and it links the LMCache, CacheGen and CacheBlend papers. The README says its development is supported in part by Tensormesh.

## Features
- Tiered KV cache offloading across GPU, CPU memory, disk and remote backends
- KV cache reuse across requests, sessions and engines
- Non-prefix KV reuse with CacheBlend
- Standalone multiprocess server mode
- In-process deployment mode
- Prefill-decode disaggregation KV transfer over NVLink, RDMA or TCP
- Pluggable storage and transport backends
- Pluggable KV transformation via SERDE interface
- KV cache observability metrics
- KV cache compression
- Multi-node P2P CPU memory sharing
- CLI and benchmarking tools

## Integrations
vLLM, NVIDIA Dynamo, Redis, Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, GDS, CoreWeave AI Object Storage, Google Kubernetes Engine, AMD Instinct GPUs, PyTorch

## Platforms
LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v0.5.5

## Links
- Website: https://lmcache.ai/
- Documentation: https://docs.lmcache.ai/
- Repository: https://github.com/LMCache/LMCache
- EveryDev.ai: https://www.everydev.ai/tools/lmcache