# AKV

> Cache engineering for LLM agents that drops and repositions KV cache entries through agent-side policies and an SGLang-based inference system.

AKV is the official implementation of the paper "Cache Engineering for Agents" from the thu-nics organization. It brings KV cache engineering to LLM agents by pairing agent-side policies that decide what to drop and when to compact token positions with an SGLang-based inference system that executes those operations while reusing compatible KV cache.

## What It Is

AKV has two components. `akv.agent` is the algorithm side: it decides which messages to drop from the KV cache and when to reposition retained tokens. `akv.system` is the execution side: it carries out cache operations through SGLang's OpenAI-compatible Chat Completions API. It is released under the Apache 2.0 license.

## How the Drop and Reposition Workflow Works

A `CachePolicy` takes an OpenAI-compatible request payload and adds cache operation fields. `drop_message` names which messages' KV entries to drop and after which message, while `reposition` names when to compact the retained token positions. The augmented payload is sent to the AKV System server, which applies the drop first, then the reposition, and generates the next reply from the remaining context. Retained KV entries are reused without re-prefilling. Usage details report `cached_tokens`, `drop_skipped_tokens` and `repos_tokens`.

## Customizing Policies and Agent Integration

Policies separate three components: a trigger (when to drop), a selector (what to drop), and a reposition rule. Built-in options cover turn-based, token-based and model-based approaches, and custom functions can be supplied. A `CacheTool` interface such as `DropContextTool` lets the agent schedule drops itself through tool calls. To integrate with an existing agent harness, pass the final serialized payload through `CachePolicy.process` before sending it to the system.

## Supported Models and Setup

Drop and Reposition are enabled for text-model architectures including Qwen, Qwen1.5/2/2.5, Qwen3 (including MoE variants), GPT-OSS and the MiniMax-M2 family. Setup requires Rust and Cargo and a Python 3.12 environment installed with uv. The quick start launches the server with `python -m akv.system.launch_server` and the repo includes a demo script.

## Features
- Agent-side CachePolicy with trigger, selector and reposition components
- Drop operation removes selected messages' KV entries
- Reposition compacts retained token positions
- KV cache reuse without re-prefilling
- Turn-based, token-based and model-based built-in policies
- CacheTool interface for agent-managed cache via tool calls
- OpenAI-compatible Chat Completions API through SGLang
- Usage reporting of cached, drop-skipped and repositioned tokens
- Custom policy functions

## Integrations
SGLang, OpenAI-compatible Chat Completions API, Qwen, GPT-OSS, MiniMax-M2

## Platforms
CLI, DEVELOPER_SDK, API

## Pricing
Open Source

## Links
- Website: https://github.com/thu-nics/AKV
- Documentation: https://github.com/thu-nics/AKV#readme
- Repository: https://github.com/thu-nics/AKV
- EveryDev.ai: https://www.everydev.ai/tools/akv