kvcached

kvcached (KV cache daemon) is an open-source library from the ovg-project organization for LLM serving and training on shared GPUs. It brings OS-style virtual memory to KV caches so serving engines can reserve virtual address space up front and back it with physical GPU memory only when the cache is in use. The latest release listed is v0.1.6, and it installs as a plugin into existing SGLang or vLLM environments.

What It Is

kvcached is a KV cache memory-management layer for LLM inference engines. It decouples GPU virtual addressing from physical memory allocation, so memory is allocated on demand and reclaimed as load changes, rather than being statically reserved at engine startup. When kvcached is enabled, the README says there is no need to set a GPU memory utilization limit, because kvcached manages memory automatically.

How It Works in Practice

Users enable it with environment variables (ENABLE_KVCACHED and KVCACHED_AUTOPATCH) on an existing SGLang or vLLM setup. Documented capabilities include:

• Elastic allocation and reclamation of KV memory
• Memory limits set through the kvcached CLI or a revisioned production-control API
• A frontend router with sleep mode for idle models
• Automatic prefix caching for vLLM and RadixCache for SGLang
• Pipeline parallelism
• MLA, sliding-window and hybrid attention models
• AMD ROCm support, which defaults to a non-contiguous KV layout

The kvctl and kvtop tools list and manage control files.

Use Cases and Benchmarks

The project lists multi-LLM serving, serverless LLM deployments, compound AI pipelines, and colocating inference with other GPU workloads such as training or fine-tuning. In its own benchmark of three Llama-3.1-8B models on an A100-80G with intermittent peaks, the project reports a 2-28x TTFT reduction versus current serving engines.

Setup Path

Prerequisites are Python 3.10-3.13, PyTorch 2.10 or later, and SGLang or vLLM. Installation is through pip from PyPI, from source, or via prebuilt Docker images for SGLang, vLLM and development.