# KTransformers

> Open-source framework for CPU-GPU heterogeneous inference and fine-tuning of large MoE language models.

KTransformers is a research project from MADSys Lab at Tsinghua University, Approaching.AI, 9#AISoft and community contributors. It focuses on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project exposes two capabilities from its kt-kernel source tree: high-performance inference serving and supervised fine-tuning (SFT) with LlamaFactory. It is released under the Apache License 2.0.

## What It Is

KTransformers is a framework for running and fine-tuning very large Mixture-of-Experts (MoE) models on hardware with limited GPU memory. It places hot experts on the GPU and cold experts on the CPU, using optimized CPU kernels. The inference side (kt-kernel) provides Intel AMX and AVX512/AVX2 kernels for INT4/INT8 quantized inference, NUMA-aware memory management for MoE, CPU-side INT4/INT8 weights and GPU-side GPTQ support. It offers a Python API for integration with SGLang and other frameworks.

## Fine-Tuning With LlamaFactory

The SFT capability integrates with LlamaFactory for fine-tuning ultra-large MoE models such as DeepSeek-V3/R1. It supports CPU/GPU hybrid fine-tuning with INT8/INT4 quantization, LoRA and full-parameter training, and DPO. The documentation includes a cookbook covering hardware checks, installation, BF16/FP8/INT8 recipes, resource planning and troubleshooting. The project states 6-12x training speedup over ZeRO-Offload in benchmarked MoE SFT workloads. A KT-FT paper describes the underlying scheduling approach.

## Model and Hardware Coverage

The changelog lists tutorials and support for models including DeepSeek-V4-Flash, Kimi K2.5/K2.6, GLM-5 series, MiniMax models, Qwen3-Next and others. Hardware coverage listed includes Intel AMX, AVX512/AVX2 CPUs, AMD GPUs via ROCm, Intel Arc GPUs and Ascend NPUs. The README reports one example of DeepSeek-R1-0528 (FP8) reaching 227.85 tokens/s total throughput on 8×L20 GPUs with a Xeon Gold 6454S.

## Current Status

The latest listed release is v0.7.1. The original integrated framework has been archived to an archive directory, and the project now organizes its documentation around kt-kernel inference and SFT.

## Features
- CPU-GPU heterogeneous inference for MoE models
- Intel AMX and AVX512/AVX2 optimized kernels
- NUMA-aware MoE memory management
- INT4/INT8 quantized CPU weights and GPU GPTQ support
- Heterogeneous expert placement and CPU-GPU expert scheduling
- LoRA and full-parameter MoE fine-tuning
- DPO fine-tuning
- 3-layer GPU-CPU-Disk prefix cache reuse
- kt-cli command line tool
- Python API for SGLang integration

## Integrations
SGLang, LlamaFactory, Ascend NPU, AMD ROCm, Intel Arc GPU

## Platforms
WINDOWS, LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v0.7.1

## Links
- Website: https://kvcache-ai.github.io/ktransformers/
- Documentation: https://kvcache-ai.github.io/ktransformers/
- Repository: https://github.com/kvcache-ai/ktransformers
- EveryDev.ai: https://www.everydev.ai/tools/ktransformers