# ExLlamaV3

> An optimized quantization and inference library for running LLMs locally on modern consumer GPUs.

ExLlamaV3 is an open-source inference and quantization library from the turboderp-org organization for running local large language models on modern consumer GPUs. It introduces the EXL3 quantization format, based on QTIP, and supports tensor-parallel and expert-parallel inference. The latest release listed on the repository is v1.6.1.

## What It Is

ExLlamaV3 is a Python library that loads Hugging Face-format models and runs them locally, with a converter that turns those models into the EXL3 format. It targets consumer hardware setups where GPU memory is limited. Example scripts, including a CLI chatbot, show how to use the backend and generator.

## Quantization and Conversion

EXL3 is a streamlined variant of QTIP from Cornell RelaxML. The converter needs only an input model in HF format and a target bitrate. It computes Hessians on the fly and uses a fused Viterbi kernel, so a model is converted in a single step. The README says this takes a couple of minutes for smaller models and up to a few hours for 70B+ models on a single high-end consumer GPU. Conversion runs can be resumed, and a working directory holds checkpoints and quantized tensors. The library also supports 2-8 bit cache quantization.

## Inference Features

- Continuous, dynamic batching and speculative decoding.
- Multimodal support for several model families.
- Flexible tensor-parallel and expert-parallel inference.
- CPU offloading so large MoE models can run with limited GPU resources, with AVX2 and AVX512 support.

## Model Support and Integrations

The README lists many architectures, including Llama, Mistral, Mixtral, Qwen, Gemma, GLM, Deepseek, GPT-OSS, Command-R and Phi, with several marked multimodal. A Transformers plugin is provided. TabbyAPI is the recommended server backend, offering an OpenAI-compatible API, HF model downloading, embedding model support and HF Jinja2 chat templates.

## Setup Path

Installation requires PyTorch with CUDA 12.4 or later, which pip does not install automatically. Options are prebuilt wheels from the releases page, a PyPI install that compiles the extension (needing the CUDA toolkit and build tools), or a source build with uv or pip. On Windows, the triton-windows package is also required.

## Features
- EXL3 quantization based on QTIP
- 2-8 bit cache quantization
- Tensor-parallel and expert-parallel inference
- CPU offloading for MoE models with AVX2/AVX512
- Continuous dynamic batching
- Speculative decoding
- Multimodal model support
- Single-step model conversion with resumable jobs
- Transformers plugin
- CLI chatbot example

## Integrations
TabbyAPI, Hugging Face Transformers, PyTorch, FlashAttention, Hugging Face Hub

## Platforms
WINDOWS, LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v1.6.1

## Links
- Website: https://github.com/turboderp-org/exllamav3
- Repository: https://github.com/turboderp-org/exllamav3
- EveryDev.ai: https://www.everydev.ai/tools/exllamav3