Swiftlet

Swiftlet is an open-source Swift and Metal runtime for the Qwen3-Next and Qwen3.5/3.6 Mixture-of-Experts model family, built by leonickson1. It keeps only the small dense core of a model in memory and streams routed expert weights from SSD on demand, so 35B and 80B models can run on Macs and iPhones. The project ships as a Swift package, a CLI, an OpenAI-compatible local server, and an iOS app integration.

What It Is

Swiftlet is a local inference engine for Apple Silicon, targeting macOS 14+ and iOS 17+. These models activate only about 3B parameters per token, so Swiftlet keeps the dense weights resident (attention, DeltaNet projections, routers, shared experts, embeddings) and fetches the needed experts from a .qpack container. The whole forward pass runs on Metal with runtime-compiled shaders, so no Metal toolchain is needed at build time.

How Expert Streaming Works

A repacker converts MLX checkpoints into a .qpack container where routed experts sit in fixed-stride blobs, so fetching one expert is a single pread from SSD rather than an mmap. Hot experts are held in a bounded cache with LFU plus recency eviction. About 75 percent of layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so those layers have no growing KV cache.

The README reports, for an M5 Mac, peak RAM of about 2.6 GB for the 4-bit Qwen3.6-35B-A3B and about 4.3 GB for the 4-bit Qwen3-Next-80B-A3B, with decode speeds of roughly 7 to 11 and 4.5 to 5 tokens per second respectively. The 35B also runs on an iPhone 17 at about 1 tok/s. Model weights are downloaded separately and keep their own terms.

Ways to Use It

• Swift package: add SwiftletCore to a macOS or iOS app and use SwiftletSession for streaming chat, conversation caching, sampling controls, cancellation, and memory-pressure handling.
• CLI: swiftlet chat and swiftlet generate for local use and benchmarking, plus swiftlet-repack to build containers from MLX checkpoints or download them from Hugging Face or a mirror.
• Server: swiftlet-server exposes the OpenAI chat-completions API on loopback.
• App: Priv AI on iOS embeds SwiftletCore, and its source is published separately.

Current Status and Tradeoffs

The README describes the runtime as working end to end with validated output, with current focus on kernel speed. Prefill runs at roughly decode speed, so long system prompts are slow, and batched prefill is the top roadmap item. The author notes the models chat and write like large models but recall facts like small ones. Layers are validated against mlx-lm reference implementations.