# AirLLM

> AirLLM enables inference of 70B+ large language models on a single 4GB GPU by streaming one layer at a time, without quantization, distillation, or pruning.

AirLLM is an open-source Python library created by Gavin Li that dramatically reduces the GPU memory required to run large language models at inference time. By decomposing a model into individual layers and streaming them one at a time through the GPU, AirLLM makes it possible to run models that would otherwise require dozens of gigabytes of VRAM on consumer-grade hardware with as little as 4GB. The project is licensed under Apache 2.0 and is available on GitHub and PyPI.

## What It Is

AirLLM is a layer-streaming inference engine for open-weight LLMs. Instead of loading an entire model into GPU memory, it splits the model into per-layer shards saved to disk, then loads and computes each layer sequentially. The VRAM requirement therefore depends on the size of a single layer rather than the total model size. This approach requires no quantization, distillation, or pruning to achieve its memory savings, though optional block-wise quantization compression (4-bit or 8-bit) is available for up to 3x inference speed improvement with minimal accuracy loss.

## How the Layer-Streaming Architecture Works

The core trick is straightforward: on first run, AirLLM decomposes the downloaded Hugging Face model into layer-wise shards and saves them to disk. During inference, it loads one layer at a time onto the GPU, runs the forward pass, then moves to the next. For sparse Mixture-of-Experts (MoE) models like DeepSeek-V3 and Kimi K3, it goes further — streaming only the experts a given token actually routes to, rather than an entire layer. This is why MoE models can run in even less VRAM than their parameter count would suggest:

- Qwen3-235B (MoE): ~3 GB VRAM
- DeepSeek-V3 671B: ~12 GB VRAM
- Kimi K3 2.8T: ~3.72 GB VRAM (measured on RTX 6000 Ada)
- Llama 3.x 70B (full precision): ~4 GB VRAM
- Llama 3.1 405B: ~8 GB VRAM

Prefetching overlaps disk loading with GPU compute to reduce idle time, and the library exposes a `profiling_mode` flag to measure time consumption per layer.

## Supported Models and Setup Path

AirLLM works with virtually every popular open-weight LLM family via a single `AutoModel.from_pretrained(huggingface_repo_id)` call. Supported families include Llama (2/3/3.1/3.3/4), Qwen (1/2/2.5/3 including MoE and FP8), DeepSeek (V2/V3/R1), Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi. Installation is a single pip command (`pip install airllm`). MacOS with Apple Silicon is supported via the `mlx` backend. CPU inference is also supported as of v2.10.1.

Key configuration options at initialization:
- `compression`: `'4bit'` or `'8bit'` for block-wise quantization speed-up
- `layer_shards_saving_path`: alternate path for the split model shards
- `hf_token`: for gated Hugging Face models
- `delete_original`: removes the original downloaded model to save disk space
- `prefetching`: overlaps loading and compute (on by default)

## Update: v3.1.0 — Kimi K3 (2.8T) Support

The latest release, v3.1.0 (published July 29, 2026), adds support for Kimi K3, described in the README as the largest open-source model released to date at 2.8 trillion parameters. The README reports it runs in 3.72 GB of VRAM on a single RTX 6000 Ada by streaming one expert at a time. K3 requires `compressed-tensors`, `flash-attn`, a CUDA 12 build of PyTorch, and `transformers` 4.56.x. The prior v3.0 release added FP8 model support and unified the API under a single `AutoModel` class covering DeepSeek-V3, Qwen3, Llama 4, Phi-4, and Gemma.

## Tradeoffs to Know

Layer-streaming inference is significantly slower than standard full-model GPU inference because each layer must be loaded from disk on every forward pass. The approach trades throughput for accessibility — it is best suited for researchers, hobbyists, and developers who need to run large models on hardware that would otherwise be incapable of loading them at all. Disk space is a meaningful constraint: splitting a 70B model is disk-intensive, and insufficient disk space is the most common reported error. The library does not replace production inference servers for high-throughput workloads.

## Features
- Run 70B LLMs on a single 4GB GPU without quantization
- Layer-by-layer streaming inference to minimize VRAM usage
- Support for 405B, 671B, and 2.8T parameter models on consumer GPUs
- Optional 4-bit and 8-bit block-wise quantization for up to 3x speed improvement
- AutoModel API auto-detects model type from Hugging Face repo ID
- Prefetching to overlap disk loading and GPU compute
- MacOS Apple Silicon support via mlx backend
- CPU inference support
- FP8 model support
- Per-expert streaming for sparse MoE models
- Configurable layer shard saving path
- Support for gated Hugging Face models via hf_token
- Profiling mode for time consumption analysis
- delete_original option to save disk space

## Integrations
Hugging Face Hub, PyTorch, bitsandbytes, mlx (Apple Silicon), safetensors, Llama 2/3/3.1/3.3/4, Qwen 1/2/2.5/3, DeepSeek V2/V3/R1, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, Yi, Google Colab, compressed-tensors, flash-attn

## Platforms
MACOS, LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v3.1.0

## Links
- Website: https://github.com/lyogavin/airllm
- Documentation: https://github.com/lyogavin/airllm#readme
- Repository: https://github.com/lyogavin/airllm
- EveryDev.ai: https://www.everydev.ai/tools/airllm