# AWQ (llm-awq)

> Activation-aware Weight Quantization toolkit for low-bit (INT3/INT4) compression and acceleration of LLMs and multi-modal models.

AWQ (Activation-aware Weight Quantization) is an open-source toolkit from MIT HAN Lab for compressing large language models to low-bit weights (INT3/INT4) and accelerating their inference. It supports instruction-tuned and multi-modal models, and the repository is MIT-licensed. The accompanying paper received the Best Paper Award at MLSys 2024, according to the repository.

## What It Is

AWQ is a post-training weight quantization method packaged as a Python and PyTorch codebase. It searches for quantization settings that preserve accuracy, then produces real 4-bit weights that reduce GPU memory use. The repo also ships CUDA kernels for W4A16 (4-bit weight, 16-bit activation) inference, covering both context and decoding stages, and a memory-efficient 4-bit Linear layer in PyTorch.

## Workflow and Outputs

The usage flow runs an AWQ search and saves the results, optionally evaluates the model on WikiText-2 with simulated quantization, and generates real INT4 quantized weights. The quantized model can then be loaded and evaluated with lower GPU memory use. Pre-computed search results are provided in an AWQ model zoo on Hugging Face covering Llama-1/2/3, OPT, CodeLlama, StarCoder, Vicuna, VILA, NVILA, LLaVA, Qwen-2.5 and DeepSeek-R1-Distill families. Models not in the zoo can be quantized by running the search yourself. BF16 precision is also supported.

## TinyChat and Edge Inference

TinyChat is a companion inference interface built on AWQ for running LLMs and vision-language models on cloud GPUs and resource-constrained edge devices such as NVIDIA Jetson Orin. The README reports W4A16 speedups over FP16 for Llama-3-8B on RTX 4090 (2.7x) and Jetson Orin (2.9x). TinyChat 2.0 is described as improving prefilling speed 1.5-1.7x over the previous version and includes chunk prefilling for multi-round Q&A.

## Ecosystem Integrations

The README notes AWQ integration in Hugging Face transformers, NVIDIA TensorRT-LLM, vLLM, Hugging Face TGI, LMDeploy, FastChat, Intel Neural Compressor, Amazon SageMaker containers and Google Vertex AI.

## Features
- Activation-aware weight quantization search for INT3/INT4
- Pre-computed AWQ model zoo
- Memory-efficient 4-bit Linear in PyTorch
- W4A16 CUDA kernels for context and decoding stages
- Support for instruction-tuned and multi-modal models
- BF16 precision support
- Chunk prefilling for multi-round Q&A
- TinyChat edge inference for LLMs and VLMs
- WikiText-2 evaluation of quantized models

## Integrations
Hugging Face Transformers, NVIDIA TensorRT-LLM, vLLM, Hugging Face TGI, LMDeploy, FastChat, Intel Neural Compressor, Amazon SageMaker, Google Vertex AI, VILA, LLaVA

## Platforms
LINUX, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Links
- Website: https://github.com/mit-han-lab/llm-awq
- Repository: https://github.com/mit-han-lab/llm-awq
- EveryDev.ai: https://www.everydev.ai/tools/llm-awq