AKV

AKV is the official implementation of the paper "Cache Engineering for Agents" from the thu-nics organization. It brings KV cache engineering to LLM agents by pairing agent-side policies that decide what to drop and when to compact token positions with an SGLang-based inference system that executes those operations while reusing compatible KV cache.

What It Is

AKV has two components. akv.agent is the algorithm side: it decides which messages to drop from the KV cache and when to reposition retained tokens. akv.system is the execution side: it carries out cache operations through SGLang's OpenAI-compatible Chat Completions API. It is released under the Apache 2.0 license.

How the Drop and Reposition Workflow Works

A CachePolicy takes an OpenAI-compatible request payload and adds cache operation fields. drop_message names which messages' KV entries to drop and after which message, while reposition names when to compact the retained token positions. The augmented payload is sent to the AKV System server, which applies the drop first, then the reposition, and generates the next reply from the remaining context. Retained KV entries are reused without re-prefilling. Usage details report cached_tokens, drop_skipped_tokens and repos_tokens.

Customizing Policies and Agent Integration

Policies separate three components: a trigger (when to drop), a selector (what to drop), and a reposition rule. Built-in options cover turn-based, token-based and model-based approaches, and custom functions can be supplied. A CacheTool interface such as DropContextTool lets the agent schedule drops itself through tool calls. To integrate with an existing agent harness, pass the final serialized payload through CachePolicy.process before sending it to the system.

Supported Models and Setup

Drop and Reposition are enabled for text-model architectures including Qwen, Qwen1.5/2/2.5, Qwen3 (including MoE variants), GPT-OSS and the MiniMax-M2 family. Setup requires Rust and Cargo and a Python 3.12 environment installed with uv. The quick start launches the server with python -m akv.system.launch_server and the repo includes a demo script.