Mesh LLM

Mesh LLM is an open-source project from the Mesh-LLM organization that pools GPUs and memory across machines and exposes them as a single OpenAI-compatible API at localhost. Nodes can run a model locally, route requests to a peer, or split a large model across several machines. The latest listed release is v0.78.1.

What It Is

Mesh LLM is a distributed inference runtime and CLI for running language models across homelab, business, or internet-connected hardware. Start one node, add more later, and the mesh decides whether a model runs locally, routes to a peer, or uses Skippy stage splits for models too large for a single machine. The project describes itself as experimental distributed-systems software.

How the Mesh Works

Every node exposes the same /v1 API, and requests are routed by the model field to a peer that can serve that model. For large dense models, Skippy loads package-backed layer stages: the coordinator plans contiguous layer ranges, starts downstream stages first, and publishes the stage-0 route. Layer packages contain a manifest plus GGUF fragments so each peer fetches only the pieces for its stage. Traffic between nodes, including split-model activations, is encrypted over QUIC. Published meshes are advertised via Nostr discovery, while private meshes use invite tokens.

Setup and Workflows

After installing with the install script, Homebrew on Apple Silicon, or building from source with just, run mesh-llm setup. The command mesh-llm serve --auto joins the best discovered public mesh, downloads a suitable model if needed, starts the API on port 9337, and starts the web console on port 3131. Other documented workflows include private meshes, publishing a mesh, joining by invite token, API-only clients, and launching Goose, OpenCode, Claude Code, or Pi through mesh-llm. A catalog of Hugging Face model refs shows which models have layer packages for multi-machine serving.

Extensibility and Experimental Features

A plugin system lets plugins declare capabilities in a manifest, exposed over MCP, HTTP, inference, and mesh events. Documented plugins include Flash-MoE SSD expert streaming and DwarfStar for ds4 models. An experimental Mixture-of-Agents mode sends a request with model set to mesh, fans it out to all models in the mesh, and arbitrates one reply; it requires at least two distinct models. SDKs are listed for Rust, Node.js/Electron, Swift, and Kotlin/Java.