shoehorn

shoehorn is an open-source CLI tool written in Rust that solves a problem preset quantizations ignore: your actual hardware memory. Instead of picking a fixed quantization level like Q4_K_M and hoping it fits, shoehorn measures your available VRAM, subtracts what inference needs (KV cache, compute buffers, safety margin), and solves a per-tensor mixed-precision assignment that fills the remainder to within a rounding error — routinely achieving 99.99% budget utilization, sometimes to the byte. The project is MIT-licensed and currently at v0.3.0, released on August 19, 2026.

What It Is

shoehorn is a hardware-aware LLM quantizer and runner. It takes a BF16 (or F16/F32) GGUF model from Hugging Face or a local path, computes the exact memory budget available on your machine, and uses a Lagrangian relaxation knapsack solver to assign the best quantization type to each tensor individually — spending every spare megabyte where the importance matrix says it buys the most quality. The output is a standard GGUF v3 file that any llama.cpp-compatible tool (LM Studio, ollama, llama-server) can load directly.

How the Solver Works

The pipeline runs in five stages: probe, budget, measure, solve, and write.

• Probe: On Apple Silicon, shoehorn queries Metal's recommendedMaxWorkingSetSize; on NVIDIA it uses NVML; on AMD it falls back to rocm-smi. --budget overrides the probe for cross-machine quantization.
• Budget: KV cache size is computed exactly from the model's GGUF hyperparameters (block count, KV head count, key/value lengths). A compute buffer estimate and configurable reserve are subtracted to yield the weight budget.
• Measure: Every quantizable tensor is scored across a candidate ladder (IQ2_XXS through F16) by encoding and decoding a sample of rows and accumulating imatrix-weighted squared error — the true end-to-end distortion under llama.cpp's decoder. This parallelizes across all cores.
• Solve: A multiple-choice knapsack via Lagrangian relaxation (bisect the shadow price of a byte; each tensor independently picks the candidate minimizing err + λ·bytes), followed by a greedy pass that spends remaining slack on the single-tensor upgrade with the best error-per-byte ratio.
• Write: Chosen types are re-encoded row-parallel and streamed as GGUF v3 with all source metadata preserved.

Supported Formats and Architecture

shoehorn implements its own quantizer from scratch in Rust — no llama.cpp code is linked — covering the full ladder from IQ2_XXS (~2.06 bpw) through F16, including all K-quant and IQ codebook formats. Rows divisible by 256 get the full IQ ladder; rows divisible only by 32 get a legacy fallback set. The quantizer mirrors ggml's weighted objectives exactly so output is directly comparable to llama-quantize. llama.cpp handles inference and serves as an independent correctness oracle.

Local Web UI and Discovery

Running shoehorn ui opens a local web app (default port 7788) that drives the full fit pipeline without flags. Key features of the UI include:

• A tape-measure budget gauge that fills in real time as the fit runs
• A What fits this machine? discovery view that scans Hugging Face's most-downloaded GGUF repos and ranks them by the quality your budget affords
• A Preview the mix dry-run mode that shows the per-tensor breakdown before writing
• A Measure the cost step that runs shoehorn eval and reports the perplexity delta vs. the original model
• A Chat with it button that starts llama-server and opens its chat page once the model is warm
• Previous fits remembered in ~/.cache/shoehorn/fits.json with one-click Chat buttons

Update: v0.3.0

The latest release is v0.3.0, published August 19, 2026 — six days after the repository was created on August 13, 2026. The README documents benchmark results on Qwen3-0.6B, Qwen3-14B, and Qwen3-30B-A3B (MoE), showing the solver filling budgets to 99.981%, 99.998%, and 100.000% respectively. The MoE result — filling a 16.88 GiB weight budget to 0 bytes of slack at 4.75 bpw — is highlighted as the strongest configuration measured: 30B-class quality at 50.6 tok/s on a 24 GB Mac. Prebuilt binaries are available for macOS Apple Silicon, Linux x86-64 (NVIDIA/AMD), and Windows x86-64 (NVIDIA), with a Homebrew tap for macOS.

Tradeoffs to Know

• The floor is IQ2_XXS (~2.06 bpw); IQ1_S/IQ1_M are not yet implemented.
• The compute buffer estimate is a heuristic; --calibrate replaces it with a measured value at the cost of one extra model load.
• The GPU probe covers Metal, NVML, and rocm-smi (AMD parse-based, not yet field-tested on real hardware); Intel GPUs require an explicit --budget.
• Multi-GPU boxes are budgeted for one device only.
• Windows support is compile-tested but not yet field-tested; auto-imatrix generation is skipped on Windows.