Shimmy

Shimmy is an open-source local inference server maintained by Michael A. Kuykendall. It serves GGUF models through an OpenAI-compatible API, so existing AI tools and SDKs can point at a local endpoint. It runs on Airframe, a pure-Rust WebGPU (WGSL) transformer engine, and ships as a single binary with no Python runtime or C++ toolchain.

What It Is

Shimmy is an inference server for running language models locally. You install it with Cargo, start it with a path to a GGUF file, and send requests to its OpenAI-style endpoints (chat completions, text completions, streaming, and model listing). The README states that model specs are derived automatically from GGUF metadata rather than hardcoded per model.

How the Engine Works

The README says Shimmy is the server and Airframe is the engine. Inference uses WGSL compute shaders via WebGPU, targeting NVIDIA, AMD, Intel, integrated GPUs and Apple Silicon. It documents F32 accumulation with deterministic output, where the same model, seed and parameters give the same output. YaRN RoPE scaling for extended context is controlled through the SHIMMY_MAX_CTX setting. TurboShimmy INT4 KV cache compression is described as giving about 7x lower KV-cache memory in tested configurations.

Model Support and Certification

The README lists 26 certified model/quantization combinations across 12 families, including Llama, Qwen, Phi, Gemma, DeepSeek-R1, Ministral and StarCoder2. Each passes a three-part certification of math, inference and determinism. SafeTensors files can be loaded, but full Airframe-native SafeTensors inference is listed as roadmap work.

Setup Path

Run cargo install shimmy, then shimmy serve --model-path <model.gguf> --bind 127.0.0.1:11435. The shimmy list command shows available models, and curl or any OpenAI client can query the local server. The README notes that v2 removed the llama.cpp, MLX, HuggingFace and RustChain backends.