Hypura

Hypura is a storage-tier-aware LLM inference scheduler built for Apple Silicon Macs. It solves a common problem for developers and researchers working with large language models on consumer hardware: models that exceed available memory cause swap-thrashing and out-of-memory crashes under standard inference tools like llama.cpp. Hypura addresses this by intelligently placing model tensors across three storage tiers — GPU (Metal), RAM, and NVMe — based on access patterns, bandwidth costs, and hardware capabilities.

The scheduler reads a GGUF model file, profiles the host hardware including GPU working set size, RAM capacity, and NVMe sequential read bandwidth, then solves a placement optimization that assigns every tensor to the appropriate tier. Norms and embeddings, which are small but accessed every token, get pinned to GPU. For Mixture-of-Experts architectures like Mixtral, router interception identifies which experts are selected per token and loads only the needed expert strides from NVMe, achieving a 75 percent I/O reduction. A neuron cache tracks loaded expert slices across tokens with a 99.5 percent hit rate from temporal locality. Dense FFN weights stream from NVMe through a dynamically sized pool buffer while attention and norms stay GPU-resident.

Hypura selects between three inference modes automatically. Full-resident mode runs when the model fits entirely in GPU and RAM with no NVMe I/O and zero overhead. Expert-streaming mode handles MoE models by keeping only non-expert tensors on GPU and streaming expert weights on demand. Dense FFN-streaming mode extends this approach to non-MoE models like Llama 70B by keeping attention and norms on GPU while streaming FFN tensors from NVMe with scaled prefetch lookahead.

Benchmarks on an M1 Max with 32 GB unified memory and 5.1 GB per second NVMe read speed show Qwen 2.5 14B running at 21 tokens per second in full-resident mode with zero overhead versus stock llama.cpp. A 31 GB Mixtral 8x7B achieves 2.2 tokens per second in expert-streaming mode where llama.cpp crashes with OOM. A 40 GB Llama 70B runs at 0.3 tokens per second in dense FFN-streaming mode, again where llama.cpp fails entirely.

Hypura also exposes an Ollama-compatible HTTP server, making it a drop-in replacement for any tool that speaks the Ollama protocol including OpenClaw. The server supports text completion, chat completion with NDJSON streaming, model metadata queries, and health checks. Configuration is automatic with no manual tuning of pool buffer sizes, prefetch depth, or memory budgets required.

The project is written in Rust and organized as a Cargo workspace with two crates: the main binary and library, and FFI bindings to a vendored llama.cpp built via CMake. It requires Rust 1.75 or newer and CMake to build from source. Hypura performs only read operations on the SSD during inference, generating zero write wear on the storage device.