# Distributed Llama

> Open-source C++ tool that connects multiple home devices into a cluster to accelerate LLM inference using tensor parallelism over Ethernet.

Distributed Llama is an open-source project by Bartłomiej Tadych (b4rtaz) that links several ordinary devices into a cluster for running large language models. It splits the neural network across machines using tensor parallelism and synchronizes state over Ethernet. It runs on Linux, macOS and Windows and is optimized for ARM and x86_64 AVX2 CPUs. The latest listed release is v0.17.0.

## What It Is

Distributed Llama is a local, distributed LLM inference engine written in C++. One root node loads the model and weights and sends them to worker nodes, then keeps the network state synchronized. The root node also processes its own slice of the network. Worker nodes process their slices and need no model-specific configuration. RAM usage is split across all nodes, with the root node needing slightly more than workers.

## How the Workflow Runs

A single `launch.py` command downloads a model and tokenizer and sets up the root node; this needs Python 3 and a C++ compiler. Documented models include Llama 3.1, 3.2 and 3.3, DeepSeek R1 Distill Llama 8B, and several Qwen 3 and Qwen 3.5 variants. Hugging Face models can also be converted manually. The CLI offers `dllama inference` (with a simple benchmark), `dllama chat`, `dllama worker`, and `dllama-api` to start an API server. Arguments control model path, tokenizer, worker addresses, thread count, maximum sequence length and sync precision. Guides cover Raspberry Pi and GPU setups, and the README notes experimental Vulkan support.

## Known Limitations

- The node count must be a power of two (1, 2, 4, ...).
- The maximum number of nodes equals the number of KV heads in the model.
- Only q40 models with q80 buffer-float-type and f32 models with f32 buffer-float-type are supported.

## Features
- Tensor-parallel LLM inference across multiple devices
- Root and worker node architecture
- Single-command model download and setup via launch.py
- CLI chat and inference benchmark
- API server (dllama-api)
- Hugging Face model conversion
- Experimental Vulkan GPU support
- Optimized for ARM and x86_64 AVX2 CPUs
- Raspberry Pi support

## Integrations
Hugging Face, Vulkan

## Platforms
LINUX, MACOS, WINDOWS, CLI, API

## Pricing
Open Source

## Version
v0.17.0

## Links
- Website: https://github.com/b4rtaz/distributed-llama
- Documentation: https://github.com/b4rtaz/distributed-llama/tree/main/docs
- Repository: https://github.com/b4rtaz/distributed-llama
- EveryDev.ai: https://www.everydev.ai/tools/distributed-llama
