# ExLlamaV2

> An inference library for running quantized LLMs locally on modern consumer GPUs, with EXL2 quantization and a dynamic generator.

ExLlamaV2 is an open-source Python inference library from turboderp-org for running local large language models on modern consumer GPUs. The repository is archived, and the README states that development continues on ExLlamaV3. The latest listed release is v0.3.2, published 2025-07-13.

## What It Is

ExLlamaV2 loads and runs quantized LLMs locally on consumer GPUs. It supports GPTQ models and its own EXL2 quantization format, and it includes a console chatbot and example scripts. The README names TabbyAPI as the official and recommended backend server; it provides an OpenAI-compatible API for local or remote inference.

## Dynamic Generator

From v0.1.0 onward, the library offers a dynamic generator with dynamic batching, smart prompt caching, K/V cache deduplication and a simplified API. It supports single, batched and asyncio streamed generation, along with sampling and speculative decoding. Paged attention is available through Flash Attention 2.5.7+. FP8 cache is not supported, but a Q4 cache mode is.

## EXL2 Quantization

EXL2 is based on the same optimization method as GPTQ and supports 2, 3, 4, 5, 6 and 8-bit quantization. Levels can be mixed within a model to reach any average bitrate between 2 and 8 bits per weight. A conversion script quantizes models against calibration data, and evaluation scripts are included. The author reports that Llama2 70B can run on a single 24 GB GPU at about 2.55 bits per weight with a 2048-token context.

## Setup and Integrations

Installation options are building from source, using prebuilt release wheels, or installing from PyPI. Building from source requires the CUDA Toolkit, a compiler (gcc on Linux, Visual Studio Build Tools on Windows) and a matching PyTorch version. The README lists integrations with TabbyAPI, ExUI, text-generation-webui and lollms-webui.

## Features
- Local LLM inference on consumer GPUs
- EXL2 quantization format (2-8 bits per weight)
- GPTQ model support
- Dynamic generator with dynamic batching
- Smart prompt caching and K/V cache deduplication
- Paged attention via Flash Attention 2.5.7+
- Speculative decoding
- Q4 cache mode
- Async streamed generation
- Multi-GPU inference with gpu_split
- Model conversion and evaluation scripts
- Console chatbot example

## Integrations
TabbyAPI, ExUI, text-generation-webui, lollms-webui, SillyTavern, PyTorch, Flash Attention, Hugging Face

## Platforms
WINDOWS, LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v0.3.2

## Links
- Website: https://github.com/turboderp-org/exllamav2
- Documentation: https://github.com/turboderp-org/exllamav2/blob/master/doc/dynamic.md
- Repository: https://github.com/turboderp-org/exllamav2
- EveryDev.ai: https://www.everydev.ai/tools/exllamav2