# shoehorn

> A CLI tool that quantizes LLM models to fit exactly into your available VRAM using per-tensor mixed-precision assignment, maximizing model quality within your hardware's memory budget.

shoehorn is an open-source CLI tool written in Rust that solves a problem preset quantizations ignore: your actual hardware memory. Instead of picking a fixed quantization level like Q4_K_M and hoping it fits, shoehorn measures your available VRAM, subtracts what inference needs (KV cache, compute buffers, safety margin), and solves a per-tensor mixed-precision assignment that fills the remainder to within a rounding error — routinely achieving 99.99% budget utilization, sometimes to the byte. The project is MIT-licensed and currently at v0.3.0, released on August 19, 2026.

## What It Is

shoehorn is a hardware-aware LLM quantizer and runner. It takes a BF16 (or F16/F32) GGUF model from Hugging Face or a local path, computes the exact memory budget available on your machine, and uses a Lagrangian relaxation knapsack solver to assign the best quantization type to each tensor individually — spending every spare megabyte where the importance matrix says it buys the most quality. The output is a standard GGUF v3 file that any llama.cpp-compatible tool (LM Studio, ollama, llama-server) can load directly.

## How the Solver Works

The pipeline runs in five stages: probe, budget, measure, solve, and write.

- **Probe**: On Apple Silicon, shoehorn queries Metal's `recommendedMaxWorkingSetSize`; on NVIDIA it uses NVML; on AMD it falls back to `rocm-smi`. `--budget` overrides the probe for cross-machine quantization.
- **Budget**: KV cache size is computed exactly from the model's GGUF hyperparameters (block count, KV head count, key/value lengths). A compute buffer estimate and configurable reserve are subtracted to yield the weight budget.
- **Measure**: Every quantizable tensor is scored across a candidate ladder (IQ2_XXS through F16) by encoding and decoding a sample of rows and accumulating imatrix-weighted squared error — the true end-to-end distortion under llama.cpp's decoder. This parallelizes across all cores.
- **Solve**: A multiple-choice knapsack via Lagrangian relaxation (bisect the shadow price of a byte; each tensor independently picks the candidate minimizing `err + λ·bytes`), followed by a greedy pass that spends remaining slack on the single-tensor upgrade with the best error-per-byte ratio.
- **Write**: Chosen types are re-encoded row-parallel and streamed as GGUF v3 with all source metadata preserved.

## Supported Formats and Architecture

shoehorn implements its own quantizer from scratch in Rust — no llama.cpp code is linked — covering the full ladder from IQ2_XXS (~2.06 bpw) through F16, including all K-quant and IQ codebook formats. Rows divisible by 256 get the full IQ ladder; rows divisible only by 32 get a legacy fallback set. The quantizer mirrors ggml's weighted objectives exactly so output is directly comparable to `llama-quantize`. llama.cpp handles inference and serves as an independent correctness oracle.

## Local Web UI and Discovery

Running `shoehorn ui` opens a local web app (default port 7788) that drives the full fit pipeline without flags. Key features of the UI include:

- A tape-measure budget gauge that fills in real time as the fit runs
- A **What fits this machine?** discovery view that scans Hugging Face's most-downloaded GGUF repos and ranks them by the quality your budget affords
- A **Preview the mix** dry-run mode that shows the per-tensor breakdown before writing
- A **Measure the cost** step that runs `shoehorn eval` and reports the perplexity delta vs. the original model
- A **Chat with it** button that starts llama-server and opens its chat page once the model is warm
- Previous fits remembered in `~/.cache/shoehorn/fits.json` with one-click Chat buttons

## Update: v0.3.0

The latest release is v0.3.0, published August 19, 2026 — six days after the repository was created on August 13, 2026. The README documents benchmark results on Qwen3-0.6B, Qwen3-14B, and Qwen3-30B-A3B (MoE), showing the solver filling budgets to 99.981%, 99.998%, and 100.000% respectively. The MoE result — filling a 16.88 GiB weight budget to 0 bytes of slack at 4.75 bpw — is highlighted as the strongest configuration measured: 30B-class quality at 50.6 tok/s on a 24 GB Mac. Prebuilt binaries are available for macOS Apple Silicon, Linux x86-64 (NVIDIA/AMD), and Windows x86-64 (NVIDIA), with a Homebrew tap for macOS.

## Tradeoffs to Know

- The floor is IQ2_XXS (~2.06 bpw); IQ1_S/IQ1_M are not yet implemented.
- The compute buffer estimate is a heuristic; `--calibrate` replaces it with a measured value at the cost of one extra model load.
- The GPU probe covers Metal, NVML, and rocm-smi (AMD parse-based, not yet field-tested on real hardware); Intel GPUs require an explicit `--budget`.
- Multi-GPU boxes are budgeted for one device only.
- Windows support is compile-tested but not yet field-tested; auto-imatrix generation is skipped on Windows.

## Features
- Per-tensor mixed-precision quantization via Lagrangian knapsack solver
- Hardware-aware VRAM probing (Metal, NVML, rocm-smi)
- Exact KV cache budget computation from GGUF hyperparameters
- Imatrix-weighted error scoring for quality-optimal bit allocation
- Full quantization format ladder: IQ2_XXS through F16
- Standard GGUF v3 output compatible with llama.cpp, LM Studio, ollama
- Local web UI with tape-measure budget gauge and chat interface
- Model discovery: ranks Hugging Face models by quality your budget affords
- Perplexity evaluation via shoehorn eval with baseline comparison
- Resumable Hugging Face downloads with shard-aware fetching
- Dry-run plan mode to preview per-tensor mix before encoding
- Previous fits remembered with one-click Chat buttons
- Cross-machine quantization via --budget and --target flags
- MoE model support with per-expert imatrix slicing
- Homebrew tap for macOS Apple Silicon
- Prebuilt binaries for macOS, Linux, and Windows

## Integrations
llama.cpp, llama-server, Hugging Face, LM Studio, ollama, NVML, rocm-smi, Metal (Apple), Homebrew

## Platforms
WINDOWS, MACOS, LINUX, WEB, API, CLI

## Pricing
Open Source

## Version
v0.3.0

## Links
- Website: https://notactuallytreyanastasio.github.io/shoehorn/
- Documentation: https://github.com/notactuallytreyanastasio/shoehorn/blob/master/DESIGN.md
- Repository: https://github.com/notactuallytreyanastasio/shoehorn
- EveryDev.ai: https://www.everydev.ai/tools/shoehorn