AWQ (llm-awq)

AWQ (Activation-aware Weight Quantization) is an open-source toolkit from MIT HAN Lab for compressing large language models to low-bit weights (INT3/INT4) and accelerating their inference. It supports instruction-tuned and multi-modal models, and the repository is MIT-licensed. The accompanying paper received the Best Paper Award at MLSys 2024, according to the repository.

What It Is

AWQ is a post-training weight quantization method packaged as a Python and PyTorch codebase. It searches for quantization settings that preserve accuracy, then produces real 4-bit weights that reduce GPU memory use. The repo also ships CUDA kernels for W4A16 (4-bit weight, 16-bit activation) inference, covering both context and decoding stages, and a memory-efficient 4-bit Linear layer in PyTorch.

Workflow and Outputs

The usage flow runs an AWQ search and saves the results, optionally evaluates the model on WikiText-2 with simulated quantization, and generates real INT4 quantized weights. The quantized model can then be loaded and evaluated with lower GPU memory use. Pre-computed search results are provided in an AWQ model zoo on Hugging Face covering Llama-1/2/3, OPT, CodeLlama, StarCoder, Vicuna, VILA, NVILA, LLaVA, Qwen-2.5 and DeepSeek-R1-Distill families. Models not in the zoo can be quantized by running the search yourself. BF16 precision is also supported.

TinyChat and Edge Inference

TinyChat is a companion inference interface built on AWQ for running LLMs and vision-language models on cloud GPUs and resource-constrained edge devices such as NVIDIA Jetson Orin. The README reports W4A16 speedups over FP16 for Llama-3-8B on RTX 4090 (2.7x) and Jetson Orin (2.9x). TinyChat 2.0 is described as improving prefilling speed 1.5-1.7x over the previous version and includes chunk prefilling for multi-round Q&A.

Ecosystem Integrations

The README notes AWQ integration in Hugging Face transformers, NVIDIA TensorRT-LLM, vLLM, Hugging Face TGI, LMDeploy, FastChat, Intel Neural Compressor, Amazon SageMaker containers and Google Vertex AI.