MLC LLM

MLC LLM is an open-source machine learning compiler and deployment engine for large language models, published by the mlc-ai organization under the Apache 2.0 license. Its stated mission is to let people develop, optimize, and deploy AI models natively on their own platforms. It compiles models and runs them on MLCEngine, a unified inference engine.

What It Is

MLC LLM combines ML compilation with an inference runtime. Models are compiled for a target platform and executed by MLCEngine, which exposes an OpenAI-compatible API. That API is available through a REST server, Python, JavaScript, iOS, and Android, all backed by the same engine and compiler.

Supported Hardware and Platforms

The README lists the following backends:

• Linux and Windows: Vulkan and ROCm on AMD GPUs, Vulkan and CUDA on NVIDIA GPUs, and Vulkan on Intel GPUs.
• macOS: Metal on AMD discrete GPUs, Apple GPUs, and Intel integrated GPUs.
• Web browsers: WebGPU and WASM.
• iOS and iPadOS: Metal on Apple A-series GPUs.
• Android: OpenCL on Adreno and Mali GPUs.

Getting Started

The documentation site provides installation instructions, a quick start, and an introduction. The related WebLLM project covers in-browser deployment. The README cites the TVM, TensorIR, and MetaSchedule research as the underlying techniques.