LMDeploy
Open-source toolkit for compressing, deploying, and serving large language models and vision-language models.
At a Glance
LMDeploy toolkit for compressing, deploying, and serving LLMs, released under the Apache 2.0 license for self-hosted use.
Engagement
Available On
Alternatives
Listed Nov 2026
About LMDeploy
LMDeploy is an open-source toolkit for compressing, deploying, and serving LLMs, developed by the MMRazor and MMDeploy teams under the InternLM organization. It provides offline inference pipelines and OpenAI-compatible API servers for both language and vision-language models. The latest GitHub release listed is v0.18.0.
What It Is
LMDeploy covers the path from a model checkpoint to a running inference service. Users install it with pip, load a model from HuggingFace (or ModelScope or openMind Hub via environment variables), and run batch inference through a Python pipeline or launch an API server. It also includes quantization tooling and a request distributor for multi-model, multi-machine deployments.
Inference Engines
LMDeploy ships two engines. TurboMind focuses on maximizing inference performance, while the PyTorch engine is written purely in Python to lower the barrier for developers experimenting with new features. Each supports a different set of models and data types, and the docs include a table for choosing between them. The project states that it delivers up to 1.8x higher request throughput than vLLM, using persistent batching, blocked KV cache, dynamic split and fuse, tensor parallelism, and optimized CUDA kernels.
Quantization and Serving Features
The docs cover AWQ/GPTQ weight quantization, SmoothQuant, INT4/INT8 KV cache quantization, and llm-compressor support. The project says 4-bit inference is 2.4x faster than FP16 and that quality was checked with OpenCompass. Serving documentation covers tool calling, reasoning outputs, Anthropic-compatible endpoints, LoRA serving, structured output, speculative decoding, context parallel, and production metrics.
Models and Hardware
Supported models include Llama, InternLM, Qwen, DeepSeek, GLM, Mixtral, Gemma, and Phi families, plus VLMs such as InternVL, LLaVA, Qwen-VL, CogVLM, and MiniCPM-V. Beyond NVIDIA GPUs, the docs include getting-started guides for Huawei Ascend, MetaX, and Cambricon platforms.
Community Discussions
Be the first to start a conversation about LMDeploy
Share your experience with LMDeploy, ask questions, or help others learn from your insights.
Pricing
Open Source (Apache 2.0)
LMDeploy toolkit for compressing, deploying, and serving LLMs, released under the Apache 2.0 license for self-hosted use.
- Apache License 2.0 licensed core toolkit
- Self-hosted; install via pip install lmdeploy
- Compression, deployment and serving of LLMs
- OpenAI-compatible server
- Quantization (AWQ/GPTQ, SmoothQuant, KV cache)
Capabilities
Key Features
- TurboMind and PyTorch inference engines
- Persistent (continuous) batching and blocked KV cache
- Weight-only AWQ/GPTQ and SmoothQuant quantization
- INT4/INT8 KV cache quantization
- OpenAI-compatible API server
- Anthropic-compatible endpoints
- Tool calling and reasoning outputs
- LoRA serving
- Vision-language model deployment
- Request distributor for multi-model multi-machine serving
- Structured output and speculative decoding
- Production metrics and benchmarking
- Multi-node deployment with PyTorchEngine
