GPT-QModel

GPT-QModel is an LLM quantization toolkit from ModelCloud that provides one API for calibration, quantization, evaluation, model conversion, and accelerated inference. It covers GPTQ, AWQ, ParoQuant, QQQ, GGUF, FP8, EXL3, GPTAQ, EoRA, GAR, and FOEM, and targets NVIDIA CUDA, AMD ROCm, Huawei Ascend, Intel XPU, Apple Silicon, and CPU. The latest listed release is v7.6.0, and quantized models can be served through Transformers, vLLM, and SGLang.

What It Is

GPT-QModel is a Python library for compressing large language models to low-bit formats. A typical workflow loads a Hugging Face model with a config such as GPTQConfig, runs quantize on a calibration dataset, saves the checkpoint, and reloads it for inference. Weight-only formats such as GGUF and FP8 skip calibration. An OpenAI-compatible endpoint is available through model.serve.

Quantization Workflow and Controls

Each method shares a common lifecycle of calibrate, quantize, validate, save, load, and serve, while keeping method-specific controls. Documented options include:

• Per-module dynamic overrides that change bits and group size or skip modules.
• Group Aware Reordering.
• Experimental GPTAQ and FOEM compensation.
• EoRA LoRA-based accuracy recovery for GPTQ models.
• Preprocessors such as smoothing and FP8/FP4 source decoding.
• Resumable quantization checkpoints, currently limited to Llama and Qwen3 MoE model types.
• MoE routing overrides and a default FailSafe mode for experts that receive few calibration tokens.

Hardware, Kernels, and Runtimes

The README documents kernels including Machete, Marlin, Exllama V2 and V3, AWQ GEMM/GEMV, ParoQuant CUDA/Triton, GGUF, QQQ, BitBLAS, and Torch fallbacks. It lists validation on Linux, macOS, and Windows 11. On Apple Silicon, optional MLX support enables native MLX GPTQ/AWQ quantization. SGLang loading is limited to specific GPTQ and AWQ formats. Evaluation integrates with the Evalution benchmarking toolkit. The README also lists support for a wide range of model families.

Setup Path

Install with pip install gptqmodel, optionally with extras such as vllm, sglang, bitblas, or mlx, or build from source. Some kernels are JIT-compiled on first use and rely on ninja. The README also documents a JIT cache location and options for offline or precompiled Machete use.