AirLLM

AirLLM is an open-source Python library created by Gavin Li that dramatically reduces the GPU memory required to run large language models at inference time. By decomposing a model into individual layers and streaming them one at a time through the GPU, AirLLM makes it possible to run models that would otherwise require dozens of gigabytes of VRAM on consumer-grade hardware with as little as 4GB. The project is licensed under Apache 2.0 and is available on GitHub and PyPI.

What It Is

AirLLM is a layer-streaming inference engine for open-weight LLMs. Instead of loading an entire model into GPU memory, it splits the model into per-layer shards saved to disk, then loads and computes each layer sequentially. The VRAM requirement therefore depends on the size of a single layer rather than the total model size. This approach requires no quantization, distillation, or pruning to achieve its memory savings, though optional block-wise quantization compression (4-bit or 8-bit) is available for up to 3x inference speed improvement with minimal accuracy loss.

How the Layer-Streaming Architecture Works

The core trick is straightforward: on first run, AirLLM decomposes the downloaded Hugging Face model into layer-wise shards and saves them to disk. During inference, it loads one layer at a time onto the GPU, runs the forward pass, then moves to the next. For sparse Mixture-of-Experts (MoE) models like DeepSeek-V3 and Kimi K3, it goes further — streaming only the experts a given token actually routes to, rather than an entire layer. This is why MoE models can run in even less VRAM than their parameter count would suggest:

• Qwen3-235B (MoE): ~3 GB VRAM
• DeepSeek-V3 671B: ~12 GB VRAM
• Kimi K3 2.8T: ~3.72 GB VRAM (measured on RTX 6000 Ada)
• Llama 3.x 70B (full precision): ~4 GB VRAM
• Llama 3.1 405B: ~8 GB VRAM

Prefetching overlaps disk loading with GPU compute to reduce idle time, and the library exposes a profiling_mode flag to measure time consumption per layer.

Supported Models and Setup Path

AirLLM works with virtually every popular open-weight LLM family via a single AutoModel.from_pretrained(huggingface_repo_id) call. Supported families include Llama (2/3/3.1/3.3/4), Qwen (1/2/2.5/3 including MoE and FP8), DeepSeek (V2/V3/R1), Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and Yi. Installation is a single pip command (pip install airllm). MacOS with Apple Silicon is supported via the mlx backend. CPU inference is also supported as of v2.10.1.

Key configuration options at initialization:

• compression: '4bit' or '8bit' for block-wise quantization speed-up
• layer_shards_saving_path: alternate path for the split model shards
• hf_token: for gated Hugging Face models
• delete_original: removes the original downloaded model to save disk space
• prefetching: overlaps loading and compute (on by default)

Update: v3.1.0 — Kimi K3 (2.8T) Support

The latest release, v3.1.0 (published July 29, 2026), adds support for Kimi K3, described in the README as the largest open-source model released to date at 2.8 trillion parameters. The README reports it runs in 3.72 GB of VRAM on a single RTX 6000 Ada by streaming one expert at a time. K3 requires compressed-tensors, flash-attn, a CUDA 12 build of PyTorch, and transformers 4.56.x. The prior v3.0 release added FP8 model support and unified the API under a single AutoModel class covering DeepSeek-V3, Qwen3, Llama 4, Phi-4, and Gemma.

Tradeoffs to Know

Layer-streaming inference is significantly slower than standard full-model GPU inference because each layer must be loaded from disk on every forward pass. The approach trades throughput for accessibility — it is best suited for researchers, hobbyists, and developers who need to run large models on hardware that would otherwise be incapable of loading them at all. Disk space is a meaningful constraint: splitting a 70B model is disk-intensive, and insufficient disk space is the most common reported error. The library does not replace production inference servers for high-throughput workloads.