FastDeploy

FastDeploy is an open-source inference and deployment toolkit for large language models (LLMs) and visual language models (VLMs) built on PaddlePaddle. It is maintained by the PaddlePaddle organization and licensed under Apache 2.0. The docs describe one-command serving through an OpenAI-compatible API server with vLLM interface compatibility. The latest listed release is v2.5.0.

What It Is

FastDeploy packages model serving and acceleration techniques so that LLMs and VLMs, such as ERNIE-4.5, Qwen, DeepSeek and GLM models, can be deployed on a range of hardware. The documentation covers online serving, offline inference, a CLI, quantization, and per-model best-practice guides.

Acceleration and Serving Features

The project documents load-balanced prefill/decode (PD) disaggregation with context caching and dynamic instance role switching, plus a unified KV cache transmission library that selects between NVLink and RDMA. Quantization formats include W8A16, W8A8, W4A16, W4A8, W2A16 and FP8. Other documented techniques are speculative decoding, Multi-Token Prediction, chunked prefill, prefix caching, structured outputs, reasoning output, thinking budget, graph optimization, data parallelism and multi-node deployment.

Hardware and Model Support

Installation guides cover NVIDIA GPU, KunlunXin XPU, Intel Gaudi, Hygon DCU, Enflame S60, Iluvatar CoreX and MetaX C550, and the homepage also lists Ascend NPU. Support tables mark each model and hardware combination as supported, in progress, or no plan.

Operations and Tooling

Serving documentation covers monitoring metrics, a scheduler, graceful shutdown and a load-balancing scheduling router. A CLI offers chat, complete, serve, bench, run-batch, tokenizer and environment collection commands, and an observability section covers tracing.