PowerInfer

PowerInfer is an open-source LLM inference engine for local deployment on a PC with a single consumer-grade GPU. The repository is hosted under the Tiiny-AI organization and is MIT licensed. It splits computation between GPU and CPU based on how often neurons activate, and the README reports speedups over llama.cpp on an RTX 4090.

What It Is

PowerInfer is a C++ inference engine built on ggml and llama.cpp that serves large language models locally. It is designed for consumer hardware and targets low-latency inference and serving on one GPU. Its examples, including server and batched generation, work much like llama.cpp's, except that the -ngl argument is replaced by --vram-budget.

How Activation Locality Works

The design relies on a power-law distribution of neuron activation. A small set of frequently activated "hot" neurons is preloaded onto the GPU, while input-dependent "cold" neurons are computed on the CPU. This reduces GPU memory demand and CPU-GPU data transfer. Adaptive predictors and neuron-aware sparse operators further exploit computational sparsity. The README reports an average of 13.20 tokens/s, with a peak of 29.08 tokens/s, on an RTX 4090 across various LLMs, and up to 11.69x faster than llama.cpp.

Models and Workflow

Models use a PowerInfer GGUF format that combines LLM weights with predictor weights. Weights can be downloaded from Hugging Face or converted from original weights plus predictors with the provided conversion scripts. Supported models include ReLU-sparse Llama 2 variants, ProSparse Llama 2, Falcon (ReLU) 40B, and Bamboo-7B. Models with other activation functions, such as Mistral, original Llama and Qwen, are not currently supported. INT4 (Q4_0) quantization is available.

Setup and Platforms

Building requires CMake 3.17+ and Python 3.8+. Build options cover NVIDIA GPUs (cuBLAS), AMD GPUs (ROCm/HIP), and CPU-only. The README lists testing on x86-64 CPUs with AVX2, with or without NVIDIA GPUs, on Linux and Windows, and on Apple M chips in CPU-only mode on macOS. A Metal backend and multi-GPU support are listed as not yet done.