# Soup CLI

> An open-source CLI that fine-tunes and post-trains LLMs in one command, auto-generating configs, handling quantization, and enabling 8B model training on a 4 GB GPU via layer streaming.

Soup is a free, Apache-2.0-licensed command-line tool built by MePlay, Inc. that collapses the entire LLM post-training stack into a single YAML config and one `soup train` command. It is maintained in the open by Alpamys Makazhan and targets practitioners who want to fine-tune large language models locally without cloud infrastructure or config expertise.

## What It Is

Soup is a post-training CLI for LLMs that automates the decisions most practitioners spend hours on: picking the training method, writing the config, choosing quantization, setting learning rate and epochs, deriving evaluations from your own data, and gating every model save on a SHIP / DON'T-SHIP verdict. It supports 23 training methods, 142 ready-made recipes, and 17 quantization formats, and integrates with the broader ML ecosystem including HuggingFace, Unsloth, DeepSpeed, vLLM, Ollama, ONNX, TensorRT, Weights & Biases, SGLang, and FlashAttention.

## Layer Streaming: The Flagship Technical Capability

The headline feature introduced in the v0.72 series is **layer streaming** — a technique that keeps the frozen base model out of VRAM entirely and feeds it to the GPU one decoder layer at a time over a dedicated CUDA stream. This bounds peak VRAM to a single layer rather than the full model. The project reports the following measured results on an RTX 3050 Laptop 4 GB:

- Llama-3.1-8B-Instruct + NF4 LoRA: 119.6 tok/s at 3.32 GB peak VRAM
- Qwen2.5-3B: 264.2 tok/s at 1.76 GB peak VRAM
- DPO's reference model reuses the same streamed base with adapters disabled, adding 0 extra weight memory (measured at 0.914× the supervised SFT peak; forcing a real second instance cost +730 MB)

Layer streaming is marked BETA and currently supports transformers, text, and plain LoRA. GRPO and PPO are explicitly excluded because generation re-reads every layer per token, which streaming cannot amortize.

## The Full Post-Training Loop

Soup organizes its feature set into five stages:

- **Decide** — `soup advise` ranks PROMPT_ENG / RAG / SFT / DPO / GRPO on your data; `soup data doctor` runs 8 chat-template checks including the EOS bug that causes infinite generation
- **Train** — SFT, DPO, ORPO, SimPO, KTO, GRPO, PPO, IPO, BCO, PRM, distillation, vision, audio, TTS, unlearning; Spectrum-rank and LISA-sample layers; multi-GPU via DeepSpeed ZeRO 2/3 and FSDP; Apple MLX backend for M1–M4
- **Ship** — `soup ship` derives a SHA-pinned eval suite from your data and refuses a model that wins the task but breaks general knowledge, tool-calling, or safety; exit codes 0/2 are CI-committable
- **Operate** — `soup diagnose` scores seven failure modes; adapter lifecycle tools (diff, merge, bisect, TIES/DARE/SVD); `soup mcp serve` exposes 14 read-only tools to MCP clients like Claude Code, Cursor, and Cline
- **Secure** — compliance templates for HIPAA, SOC2, EU-AI-Act, and SR-11-7; signed ML-BOMs; SLSA-3 provenance via `soup bom` and `soup attest`

## Migration and Ecosystem Fit

Soup can convert existing configs from LLaMA-Factory, Axolotl, and Unsloth in one command (`soup migrate --from <tool> config.yaml`), lowering the switching cost for teams already invested in those tools. It auto-detects optimizer, scheduler, target modules, and batch size from the migrated config. Production traces can be ingested from Langfuse, LangSmith, Helicone, OpenPipe, OpenTelemetry, and OpenAI Stored Completions offline with no per-trace fees.

## Update: v0.72.4 — Preference Losses Over Layer Streaming

The latest release (v0.72.4, published 2026-08-03) extends layer streaming to preference alignment losses: DPO, ORPO, SimPO, and KTO now all stream. The reference model in DPO is the same streamed base with adapters switched off, so it costs no additional weights in memory. The release also fixes a bug from v0.72.0 where adapters trained with `stream_layers: true` were saved under incorrect tensor keys (`.inner.` segment), making them inert on load — users who trained on v0.72.0 are advised to re-run. The project publishes all measurement records, including failures and discarded numbers, in the `benchmarks/` directory and a citable preprint on Zenodo (DOI: 10.5281/zenodo.21771064).

## Deployment Model and Tradeoffs

Soup installs via pip (`pip install "soup-cli[train]"`) and runs entirely locally — no sign-up, no credit card, no vendor lock-in. It requires Python 3.10+ and a CUDA GPU (Apple Silicon MPS and CPU are supported experimentally). The project is built and maintained on a single 4 GB laptop, which the maintainer cites as the reason every performance number is measured rather than claimed. Multi-GPU validation above 8B and Apple Silicon benchmarks are explicitly listed as hardware-gated work items, not yet measured.

## Features
- Layer streaming: fine-tune 8B models on 4 GB GPU by streaming frozen base from RAM/NVMe
- 23 training methods: SFT, DPO, ORPO, SimPO, KTO, GRPO, PPO, IPO, BCO, PRM, distillation, and more
- 142 ready-made model recipes
- 17 quantization formats including NF4, GGUF, ONNX, TensorRT, AWQ, GPTQ, FP8, BitNet
- Auto-generated configs: batch size, optimizer, scheduler, target modules auto-detected
- soup advise: ranks PROMPT_ENG / RAG / SFT / DPO / GRPO on your data
- soup data doctor: 8 chat-template checks including EOS bug detection
- Semantic deduplication, topic mapping, canary injection, brain-rot filtering
- soup ship: SHIP/DON'T-SHIP gate over 7 offline eval suites
- Adapter lifecycle: diff, merge (TIES/DARE/SVD/CMA-ES), bisect, lock
- soup mcp serve: 14 MCP tools for Claude Code, Cursor, Cline, Continue
- Compliance templates: HIPAA, SOC2, EU-AI-Act, SR-11-7
- Signed ML-BOMs and SLSA-3 provenance via soup bom and soup attest
- Migration from LLaMA-Factory, Axolotl, Unsloth in one command
- Apple MLX backend for M1–M4 native SFT/DPO/GRPO
- Multi-GPU via DeepSpeed ZeRO 2/3, ZeRO++, and FSDP
- OpenAI- and Anthropic-compatible serving with hot-swappable adapters
- Speculative decoding with soup draft measure and distill
- Production trace ingestion from Langfuse, LangSmith, Helicone, OpenPipe, OpenTelemetry
- Docker support via GHCR image
- soup autopilot: zero-config fine-tuning from model ID + data + goal

## Integrations
HuggingFace Hub, Ollama, vLLM, DeepSpeed, Unsloth, ONNX, NVIDIA TensorRT, Weights & Biases, SGLang, FlashAttention, Apple MLX, llama.cpp, MLflow, SwanLab, Trackio, TensorBoard, Langfuse, LangSmith, Helicone, OpenPipe, OpenTelemetry, OpenAI Stored Completions, LLaMA-Factory, Axolotl

## Platforms
CLI, API, LINUX, MACOS, WINDOWS

## Pricing
Open Source

## Version
v0.72.4

## Links
- Website: https://trysoup.dev
- Documentation: https://trysoup.dev/docs
- Repository: https://github.com/MakazhanAlpamys/Soup
- EveryDev.ai: https://www.everydev.ai/tools/soup-cli