# FastDeploy

> Inference and deployment toolkit for large language models and vision-language models based on PaddlePaddle.

FastDeploy is an open-source inference and deployment toolkit for large language models (LLMs) and visual language models (VLMs) built on PaddlePaddle. It is maintained by the PaddlePaddle organization and licensed under Apache 2.0. The docs describe one-command serving through an OpenAI-compatible API server with vLLM interface compatibility. The latest listed release is v2.5.0.

## What It Is

FastDeploy packages model serving and acceleration techniques so that LLMs and VLMs, such as ERNIE-4.5, Qwen, DeepSeek and GLM models, can be deployed on a range of hardware. The documentation covers online serving, offline inference, a CLI, quantization, and per-model best-practice guides.

## Acceleration and Serving Features

The project documents load-balanced prefill/decode (PD) disaggregation with context caching and dynamic instance role switching, plus a unified KV cache transmission library that selects between NVLink and RDMA. Quantization formats include W8A16, W8A8, W4A16, W4A8, W2A16 and FP8. Other documented techniques are speculative decoding, Multi-Token Prediction, chunked prefill, prefix caching, structured outputs, reasoning output, thinking budget, graph optimization, data parallelism and multi-node deployment.

## Hardware and Model Support

Installation guides cover NVIDIA GPU, KunlunXin XPU, Intel Gaudi, Hygon DCU, Enflame S60, Iluvatar CoreX and MetaX C550, and the homepage also lists Ascend NPU. Support tables mark each model and hardware combination as supported, in progress, or no plan.

## Operations and Tooling

Serving documentation covers monitoring metrics, a scheduler, graceful shutdown and a load-balancing scheduling router. A CLI offers chat, complete, serve, bench, run-batch, tokenizer and environment collection commands, and an observability section covers tracing.

## Features
- Load-balanced PD disaggregation
- Unified KV cache transmission over NVLink/RDMA
- OpenAI-compatible API server with vLLM interface compatibility
- Quantization support (W8A16, W8A8, W4A16, W4A8, W2A16, FP8)
- Speculative decoding and Multi-Token Prediction
- Chunked prefill and prefix caching
- Structured outputs and reasoning output
- Multi-node deployment and data parallelism
- Graph optimization
- Multi-hardware support
- CLI for chat, serve, bench and batch runs
- Metrics monitoring and tracing

## Integrations
PaddlePaddle, vLLM, OpenAI API, NVIDIA GPU, KunlunXin XPU, Intel Gaudi, Hygon DCU, Ascend NPU, Enflame GCU, Iluvatar GPU, MetaX GPU

## Platforms
LINUX, CLI, API, DEVELOPER_SDK

## Pricing
Open Source

## Version
v2.5.0

## Links
- Website: https://paddlepaddle.github.io/FastDeploy/
- Documentation: https://paddlepaddle.github.io/FastDeploy/
- Repository: https://github.com/PaddlePaddle/FastDeploy
- EveryDev.ai: https://www.everydev.ai/tools/fastdeploy