EveryDev.ai
Subscribe
Home
Tools

5,901+ AI tools

  • Featured
  • Gems
  • Trending
  • New
  • Rate tools
  • Compare
  • Arena
Categories
  • Agents2972
  • Coding2248
  • Infrastructure1067
  • Projects818
  • MCP794
  • Research744
  • Analytics565
  • Marketing527
  • Design488
  • Testing476
  • Data442
  • Security412
  • Integration288
  • Communication284
  • Prompts283
  • Extensions238
  • Voice213
  • DevOps190
  • Commerce183
  • Web128
  • Learning123
  • Finance44
AI Tools by Topic
  • AI Coding Assistants
  • Agent Frameworks
  • MCP Servers
  • AI Prompt Tools
  • Vibe Coding Tools
  • AI Design Tools
  • AI Database Tools
  • AI Website Builders
  • AI Testing Tools
  • LLM Evaluations
Follow Us
  • X / Twitter
  • LinkedIn
  • Reddit
  • Discord
  • Threads
  • Bluesky
  • Mastodon
  • YouTube
  • GitHub
  • Instagram
Get Started
  • Users
  • Rate Tools
  • About
  • Editorial Standards
  • Corrections & Disclosures
  • Community Guidelines
  • Advertise
  • Contact Us
  • Newsletter
  • Submit a Tool
  • Start a Discussion
  • Write A Blog
  • Share A Build
  • Terms of Service
  • Privacy Policy
Explore with AI
  • ChatGPT
  • Gemini
  • Claude
  • Grok
  • Perplexity
Agent Experience
  • llms.txt
Theme
With AI, Everyone is a Dev. EveryDev.ai ©
    1. Home
    2. Tools
    3. LMDeploy
    LMDeploy icon

    LMDeploy

    AI Infrastructure
    Featured

    Open-source toolkit for compressing, deploying, and serving large language models and vision-language models.

    Visit Website

    At a Glance

    Pricing
    Open Source

    LMDeploy toolkit for compressing, deploying, and serving LLMs, released under the Apache 2.0 license for self-hosted use.

    Engagement

    Available On

    Linux
    Windows
    API
    CLI
    SDK

    Resources

    WebsiteDocsGitHubllms.txt

    Topics

    AI InfrastructureModel ManagementLocal Inference

    Alternatives

    AWQ (llm-awq)LittleBitSGLang
    Developer
    InternLM

    Listed Nov 2026

    About LMDeploy

    LMDeploy is an open-source toolkit for compressing, deploying, and serving LLMs, developed by the MMRazor and MMDeploy teams under the InternLM organization. It provides offline inference pipelines and OpenAI-compatible API servers for both language and vision-language models. The latest GitHub release listed is v0.18.0.

    What It Is

    LMDeploy covers the path from a model checkpoint to a running inference service. Users install it with pip, load a model from HuggingFace (or ModelScope or openMind Hub via environment variables), and run batch inference through a Python pipeline or launch an API server. It also includes quantization tooling and a request distributor for multi-model, multi-machine deployments.

    Inference Engines

    LMDeploy ships two engines. TurboMind focuses on maximizing inference performance, while the PyTorch engine is written purely in Python to lower the barrier for developers experimenting with new features. Each supports a different set of models and data types, and the docs include a table for choosing between them. The project states that it delivers up to 1.8x higher request throughput than vLLM, using persistent batching, blocked KV cache, dynamic split and fuse, tensor parallelism, and optimized CUDA kernels.

    Quantization and Serving Features

    The docs cover AWQ/GPTQ weight quantization, SmoothQuant, INT4/INT8 KV cache quantization, and llm-compressor support. The project says 4-bit inference is 2.4x faster than FP16 and that quality was checked with OpenCompass. Serving documentation covers tool calling, reasoning outputs, Anthropic-compatible endpoints, LoRA serving, structured output, speculative decoding, context parallel, and production metrics.

    Models and Hardware

    Supported models include Llama, InternLM, Qwen, DeepSeek, GLM, Mixtral, Gemma, and Phi families, plus VLMs such as InternVL, LLaVA, Qwen-VL, CogVLM, and MiniCPM-V. Beyond NVIDIA GPUs, the docs include getting-started guides for Huawei Ascend, MetaX, and Cambricon platforms.

    LMDeploy - 1

    Community Discussions

    Be the first to start a conversation about LMDeploy

    Share your experience with LMDeploy, ask questions, or help others learn from your insights.

    Pricing

    OPEN SOURCE

    Open Source (Apache 2.0)

    LMDeploy toolkit for compressing, deploying, and serving LLMs, released under the Apache 2.0 license for self-hosted use.

    • Apache License 2.0 licensed core toolkit
    • Self-hosted; install via pip install lmdeploy
    • Compression, deployment and serving of LLMs
    • OpenAI-compatible server
    • Quantization (AWQ/GPTQ, SmoothQuant, KV cache)

    Capabilities

    Key Features

    • TurboMind and PyTorch inference engines
    • Persistent (continuous) batching and blocked KV cache
    • Weight-only AWQ/GPTQ and SmoothQuant quantization
    • INT4/INT8 KV cache quantization
    • OpenAI-compatible API server
    • Anthropic-compatible endpoints
    • Tool calling and reasoning outputs
    • LoRA serving
    • Vision-language model deployment
    • Request distributor for multi-model multi-machine serving
    • Structured output and speculative decoding
    • Production metrics and benchmarking
    • Multi-node deployment with PyTorchEngine

    Integrations

    HuggingFace
    ModelScope
    openMind Hub
    OpenCompass
    VLMEvalKit
    vllm llm-compressor
    OpenAOE
    DLSlime
    Mooncake
    BentoML
    Huawei Ascend
    API Available
    View Docs

    Ratings & Reviews

    No ratings yet

    Be the first to rate LMDeploy and help others make informed decisions.

    Rate other tools you’ve used

    Developer

    InternLM

    Read more about InternLM
    GitHubX / Twitter
    1 tool in directory

    Similar Tools

    AWQ (llm-awq) icon

    AWQ (llm-awq)

    Activation-aware Weight Quantization toolkit for low-bit (INT3/INT4) compression and acceleration of LLMs and multi-modal models.

    LittleBit icon

    LittleBit

    Official implementation of LittleBit and LittleBit-2, methods for compressing large language models to sub-1-bit weights via latent factorization.

    SGLang icon

    SGLang

    Fast serving framework for large language models and vision language models with efficient inference and structured generation.

    Browse all tools

    Related Topics

    AI Infrastructure

    Infrastructure designed for deploying and running AI models.

    589 tools

    Model Management

    Tools for managing, versioning, and deploying AI models.

    144 tools

    Local Inference

    Tools and platforms for running AI inference locally without cloud dependence.

    353 tools
    Browse all topics
    Back to all toolsSuggest an edit
    ratings
    discussions
    8views