KTransformers

KTransformers is a research project from MADSys Lab at Tsinghua University, Approaching.AI, 9#AISoft and community contributors. It focuses on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project exposes two capabilities from its kt-kernel source tree: high-performance inference serving and supervised fine-tuning (SFT) with LlamaFactory. It is released under the Apache License 2.0.

What It Is

KTransformers is a framework for running and fine-tuning very large Mixture-of-Experts (MoE) models on hardware with limited GPU memory. It places hot experts on the GPU and cold experts on the CPU, using optimized CPU kernels. The inference side (kt-kernel) provides Intel AMX and AVX512/AVX2 kernels for INT4/INT8 quantized inference, NUMA-aware memory management for MoE, CPU-side INT4/INT8 weights and GPU-side GPTQ support. It offers a Python API for integration with SGLang and other frameworks.

Fine-Tuning With LlamaFactory

The SFT capability integrates with LlamaFactory for fine-tuning ultra-large MoE models such as DeepSeek-V3/R1. It supports CPU/GPU hybrid fine-tuning with INT8/INT4 quantization, LoRA and full-parameter training, and DPO. The documentation includes a cookbook covering hardware checks, installation, BF16/FP8/INT8 recipes, resource planning and troubleshooting. The project states 6-12x training speedup over ZeRO-Offload in benchmarked MoE SFT workloads. A KT-FT paper describes the underlying scheduling approach.

Model and Hardware Coverage

The changelog lists tutorials and support for models including DeepSeek-V4-Flash, Kimi K2.5/K2.6, GLM-5 series, MiniMax models, Qwen3-Next and others. Hardware coverage listed includes Intel AMX, AVX512/AVX2 CPUs, AMD GPUs via ROCm, Intel Arc GPUs and Ascend NPUs. The README reports one example of DeepSeek-R1-0528 (FP8) reaching 227.85 tokens/s total throughput on 8×L20 GPUs with a Xeon Gold 6454S.

Current Status

The latest listed release is v0.7.1. The original integrated framework has been archived to an archive directory, and the project now organizes its documentation around kt-kernel inference and SFT.