# vLLM Ascend

> Community-maintained hardware plugin that lets vLLM run LLM inference and serving on Huawei Ascend NPUs.

vLLM Ascend (vllm-ascend) is a community-maintained hardware plugin for running vLLM on Huawei Ascend NPUs, hosted under the vllm-project organization. It follows vLLM's hardware-pluggable interface so the Ascend backend stays decoupled from the core engine. The latest release listed on GitHub is v0.23.0.post1.

## What It Is

vLLM Ascend is an inference and serving backend plugin. The docs describe it as the recommended approach for supporting the Ascend backend within the vLLM community, built on the hardware-pluggable RFC. With it, Transformer-like, Mixture-of-Experts, embedding and multi-modal models can run on Ascend NPUs.

## Models and Features

The documentation includes model tutorials for families such as Qwen, DeepSeek, GLM, Kimi, MiniMax, Gemma and others, plus embedding, reranker and OCR/ASR models. The user guide covers graph mode, quantization, LoRA, structured output, speculative decoding, sleep mode, and several parallelism modes (pipeline, context, sequence). It also documents KV cache pooling and offloading, prefill/decode disaggregation (for example via Mooncake), expert parallelism load balancing, and RL usage.

## Setup Path

The README lists the prerequisites: Linux, Atlas 800I A2/A3 inference and Atlas A2/A3 training hardware (Atlas 300I Duo is marked experimental), Python 3.10 to below 3.13, CANN 9.1.0, PyTorch 2.10.0 with TorchNPU, and a vLLM version matching vllm-ascend. Quick Start and Installation guides exist for each documented release. Deployment guides cover MindIE Motor and Volcano Kthena. Evaluation guides cover EvalScope, lm_eval, AISBench and OpenCompass.

## Release and Branch Model

The main branch tracks vLLM main, while release branches are created alongside vLLM releases. The README lists v0.23.0 as the latest stable version and v0.30.0rc1 and v0.26.0rc1 as release candidates. The project is licensed under Apache 2.0.

## Features
- Hardware-pluggable vLLM backend for Ascend NPU
- Support for Transformer, MoE, embedding and multi-modal models
- Graph mode
- Quantization
- LoRA
- Structured output
- Speculative decoding
- Pipeline, context and sequence parallelism
- KV cache pooling and CPU offloading
- Prefill/decode disaggregation via Mooncake
- Expert parallelism load balancer
- Sleep mode
- Reinforcement learning support
- Evaluation guides (EvalScope, lm_eval, AISBench, OpenCompass)

## Integrations
vLLM, Ascend CANN, PyTorch, TorchNPU, Mooncake, Ray, MindIE Motor, Volcano Kthena, LMCache, LLaMA-Factory, EvalScope, OpenCompass

## Platforms
LINUX, WEB, API, DEVELOPER_SDK, CLI

## Pricing
Open Source

## Version
v0.23.0.post1

## Links
- Website: https://docs.vllm.ai/projects/ascend
- Documentation: https://docs.vllm.ai/projects/ascend/en/latest/
- Repository: https://github.com/vllm-project/vllm-ascend
- EveryDev.ai: https://www.everydev.ai/tools/vllm-ascend