ExLlamaV3

ExLlamaV3 is an open-source inference and quantization library from the turboderp-org organization for running local large language models on modern consumer GPUs. It introduces the EXL3 quantization format, based on QTIP, and supports tensor-parallel and expert-parallel inference. The latest release listed on the repository is v1.6.1.

What It Is

ExLlamaV3 is a Python library that loads Hugging Face-format models and runs them locally, with a converter that turns those models into the EXL3 format. It targets consumer hardware setups where GPU memory is limited. Example scripts, including a CLI chatbot, show how to use the backend and generator.

Quantization and Conversion

EXL3 is a streamlined variant of QTIP from Cornell RelaxML. The converter needs only an input model in HF format and a target bitrate. It computes Hessians on the fly and uses a fused Viterbi kernel, so a model is converted in a single step. The README says this takes a couple of minutes for smaller models and up to a few hours for 70B+ models on a single high-end consumer GPU. Conversion runs can be resumed, and a working directory holds checkpoints and quantized tensors. The library also supports 2-8 bit cache quantization.

Inference Features

• Continuous, dynamic batching and speculative decoding.
• Multimodal support for several model families.
• Flexible tensor-parallel and expert-parallel inference.
• CPU offloading so large MoE models can run with limited GPU resources, with AVX2 and AVX512 support.

Model Support and Integrations

The README lists many architectures, including Llama, Mistral, Mixtral, Qwen, Gemma, GLM, Deepseek, GPT-OSS, Command-R and Phi, with several marked multimodal. A Transformers plugin is provided. TabbyAPI is the recommended server backend, offering an OpenAI-compatible API, HF model downloading, embedding model support and HF Jinja2 chat templates.

Setup Path

Installation requires PyTorch with CUDA 12.4 or later, which pip does not install automatically. Options are prebuilt wheels from the releases page, a PyPI install that compiles the extension (needing the CUDA toolkit and build tools), or a source build with uv or pip. On Windows, the triton-windows package is also required.