WebLLM

WebLLM is an open-source, high-performance in-browser LLM inference engine developed by the MLC AI community, initiated by members from CMU Catalyst, UW SAMPL, SJTU, OctoML, and the MLC community. It brings language model inference directly into web browsers using WebGPU for hardware acceleration, eliminating the need for any server-side processing. The project is licensed under Apache 2.0 and is available as an npm package (@mlc-ai/web-llm), making it straightforward to embed into web applications.

What It Is

WebLLM is a TypeScript/JavaScript library that enables developers to run open-source large language models entirely client-side inside a browser tab. It leverages WebGPU for GPU-accelerated inference and WebAssembly for model computation, meaning all data stays on the user's device. It is a companion project to MLC LLM, which targets universal LLM deployment across hardware environments, and reuses MLC's model artifact and build flow.

OpenAI API Compatibility

One of WebLLM's defining design choices is full compatibility with the OpenAI Chat Completions API. Developers can use the same API surface they already know — including streaming, JSON-mode, logit-level control, seeding, and function-calling (work in progress) — but run it against open-source models locally in the browser. This means existing OpenAI-based application code can be adapted to run fully client-side with minimal changes.

Supported Models and Custom Integration

WebLLM natively supports a broad range of model families:

• Llama: Llama 3, Llama 2, Hermes-2-Pro-Llama-3
• Phi: Phi 3, Phi 2, Phi 1.5
• Gemma: Gemma-2B
• Mistral: Mistral-7B-v0.3 and several Hermes/NeuralHermes variants
• Qwen (通义千问): Qwen2 0.5B, 1.5B, 7B

Beyond built-in models, developers can compile and deploy custom models in MLC format by following the MLC LLM documentation. Custom model integration requires specifying a model URL (weights and metadata) and a model_lib URL (WebAssembly file), both of which are configurable in the engine.

Architecture and Worker Support

WebLLM is designed with a modular architecture that separates heavy computation from UI threads. It supports three worker deployment patterns:

• Dedicated Web Worker: Offloads inference to a separate thread, keeping the UI responsive.
• Service Worker: Persists the model across page visits and supports offline use; includes heartbeat-based keep-alive logic to manage the browser-managed lifecycle.
• Chrome Extension: Enables building browser extensions that run LLMs in the background, with examples for both basic and WebGPU service worker-backed extensions.

Model weights are cached in the browser using one of four configurable backends: the browser Cache API (default), IndexedDB, Origin Private File System (OPFS), or an experimental Cross-Origin Storage extension for Chrome.

Setup Path

WebLLM can be installed via npm, yarn, or pnpm (@mlc-ai/web-llm), or imported directly via CDN using jsDelivr for use on platforms like JSFiddle, Codepen, and Scribbler. The core interface is MLCEngine, instantiated via the CreateMLCEngine() factory function. Model loading is asynchronous and requires a download on first use; subsequent loads are served from the browser cache.

Update: v0.2.85

The latest release is v0.2.85, published on September 8, 2026, according to the GitHub repository. The project remains actively maintained with 19,000+ stars and over 1,300 forks on GitHub. Recent additions include structured JSON generation implemented in the WebAssembly model library, SRI (Subresource Integrity) hash verification for model artifacts, and support for the experimental Cross-Origin Storage cache backend. A research paper describing the system was published on arXiv (arXiv:2412.15803).