Distributed Llama
Open-source C++ tool that connects multiple home devices into a cluster to accelerate LLM inference using tensor parallelism over Ethernet.
At a Glance
Distributed Llama core software released under the MIT license; free to use, copy, modify, and distribute.
Engagement
Available On
Alternatives
Listed Nov 2026
About Distributed Llama
Distributed Llama is an open-source project by Bartłomiej Tadych (b4rtaz) that links several ordinary devices into a cluster for running large language models. It splits the neural network across machines using tensor parallelism and synchronizes state over Ethernet. It runs on Linux, macOS and Windows and is optimized for ARM and x86_64 AVX2 CPUs. The latest listed release is v0.17.0.
What It Is
Distributed Llama is a local, distributed LLM inference engine written in C++. One root node loads the model and weights and sends them to worker nodes, then keeps the network state synchronized. The root node also processes its own slice of the network. Worker nodes process their slices and need no model-specific configuration. RAM usage is split across all nodes, with the root node needing slightly more than workers.
How the Workflow Runs
A single launch.py command downloads a model and tokenizer and sets up the root node; this needs Python 3 and a C++ compiler. Documented models include Llama 3.1, 3.2 and 3.3, DeepSeek R1 Distill Llama 8B, and several Qwen 3 and Qwen 3.5 variants. Hugging Face models can also be converted manually. The CLI offers dllama inference (with a simple benchmark), dllama chat, dllama worker, and dllama-api to start an API server. Arguments control model path, tokenizer, worker addresses, thread count, maximum sequence length and sync precision. Guides cover Raspberry Pi and GPU setups, and the README notes experimental Vulkan support.
Known Limitations
- The node count must be a power of two (1, 2, 4, ...).
- The maximum number of nodes equals the number of KV heads in the model.
- Only q40 models with q80 buffer-float-type and f32 models with f32 buffer-float-type are supported.
Community Discussions
Be the first to start a conversation about Distributed Llama
Share your experience with Distributed Llama, ask questions, or help others learn from your insights.
Pricing
Open Source (MIT)
Distributed Llama core software released under the MIT license; free to use, copy, modify, and distribute.
- MIT License
- Connect home devices into a cluster to accelerate LLM inference
- Supports Linux, macOS, and Windows
- Optimized for ARM and x86_64 AVX2 CPUs
- Commands: dllama inference, chat, worker, dllama-api
Capabilities
Key Features
- Tensor-parallel LLM inference across multiple devices
- Root and worker node architecture
- Single-command model download and setup via launch.py
- CLI chat and inference benchmark
- API server (dllama-api)
- Hugging Face model conversion
- Experimental Vulkan GPU support
- Optimized for ARM and x86_64 AVX2 CPUs
- Raspberry Pi support
