# Mesh LLM

> Open-source distributed LLM inference that pools GPUs across machines and serves them through one OpenAI-compatible API.

Mesh LLM is an open-source project from the Mesh-LLM organization that pools GPUs and memory across machines and exposes them as a single OpenAI-compatible API at localhost. Nodes can run a model locally, route requests to a peer, or split a large model across several machines. The latest listed release is v0.78.1.

## What It Is

Mesh LLM is a distributed inference runtime and CLI for running language models across homelab, business, or internet-connected hardware. Start one node, add more later, and the mesh decides whether a model runs locally, routes to a peer, or uses Skippy stage splits for models too large for a single machine. The project describes itself as experimental distributed-systems software.

## How the Mesh Works

Every node exposes the same /v1 API, and requests are routed by the model field to a peer that can serve that model. For large dense models, Skippy loads package-backed layer stages: the coordinator plans contiguous layer ranges, starts downstream stages first, and publishes the stage-0 route. Layer packages contain a manifest plus GGUF fragments so each peer fetches only the pieces for its stage. Traffic between nodes, including split-model activations, is encrypted over QUIC. Published meshes are advertised via Nostr discovery, while private meshes use invite tokens.

## Setup and Workflows

After installing with the install script, Homebrew on Apple Silicon, or building from source with just, run mesh-llm setup. The command mesh-llm serve --auto joins the best discovered public mesh, downloads a suitable model if needed, starts the API on port 9337, and starts the web console on port 3131. Other documented workflows include private meshes, publishing a mesh, joining by invite token, API-only clients, and launching Goose, OpenCode, Claude Code, or Pi through mesh-llm. A catalog of Hugging Face model refs shows which models have layer packages for multi-machine serving.

## Extensibility and Experimental Features

A plugin system lets plugins declare capabilities in a manifest, exposed over MCP, HTTP, inference, and mesh events. Documented plugins include Flash-MoE SSD expert streaming and DwarfStar for ds4 models. An experimental Mixture-of-Agents mode sends a request with model set to mesh, fans it out to all models in the mesh, and arbitrates one reply; it requires at least two distinct models. SDKs are listed for Rust, Node.js/Electron, Swift, and Kotlin/Java.

## Features
- Pools GPUs and memory across machines behind one OpenAI-compatible API
- Routes requests by model field to the peer serving that model
- Skippy layer-package stage splits for models too large for one machine
- QUIC end-to-end encrypted peer transport
- Public mesh discovery via Nostr and private invite-token meshes
- Web console and management API
- Plugin architecture with manifests exposing MCP, HTTP, and inference capabilities
- Experimental Mixture-of-Agents gateway (model: mesh)
- Model catalog with Hugging Face refs and layer packages
- Launchers for Goose, OpenCode, Claude Code, and Pi
- SDKs for Rust, Node.js/Electron, Swift, and Kotlin/Java
- Local model-only serving mode for development

## Integrations
Goose, OpenCode, Claude Code, Pi, VS Code, Hugging Face, llama.cpp, Nostr, Flash-MoE, DwarfStar, OpenAI clients

## Platforms
MACOS, LINUX, WINDOWS, CLI, API, DEVELOPER_SDK

## Pricing
Open Source

## Version
v0.78.1

## Links
- Website: http://meshllm.cloud/
- Documentation: http://meshllm.cloud/docs/
- Repository: https://github.com/Mesh-LLM/mesh-llm
- EveryDev.ai: https://www.everydev.ai/tools/mesh-llm