# Swiftlet

> A Swift and Metal runtime that runs 35B and 80B Qwen MoE models on Apple devices by streaming expert weights from storage.

Swiftlet is an open-source Swift and Metal runtime for the Qwen3-Next and Qwen3.5/3.6 Mixture-of-Experts model family, built by leonickson1. It keeps only the small dense core of a model in memory and streams routed expert weights from SSD on demand, so 35B and 80B models can run on Macs and iPhones. The project ships as a Swift package, a CLI, an OpenAI-compatible local server, and an iOS app integration.

## What It Is

Swiftlet is a local inference engine for Apple Silicon, targeting macOS 14+ and iOS 17+. These models activate only about 3B parameters per token, so Swiftlet keeps the dense weights resident (attention, DeltaNet projections, routers, shared experts, embeddings) and fetches the needed experts from a `.qpack` container. The whole forward pass runs on Metal with runtime-compiled shaders, so no Metal toolchain is needed at build time.

## How Expert Streaming Works

A repacker converts MLX checkpoints into a `.qpack` container where routed experts sit in fixed-stride blobs, so fetching one expert is a single `pread` from SSD rather than an mmap. Hot experts are held in a bounded cache with LFU plus recency eviction. About 75 percent of layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so those layers have no growing KV cache.

The README reports, for an M5 Mac, peak RAM of about 2.6 GB for the 4-bit Qwen3.6-35B-A3B and about 4.3 GB for the 4-bit Qwen3-Next-80B-A3B, with decode speeds of roughly 7 to 11 and 4.5 to 5 tokens per second respectively. The 35B also runs on an iPhone 17 at about 1 tok/s. Model weights are downloaded separately and keep their own terms.

## Ways to Use It

- Swift package: add SwiftletCore to a macOS or iOS app and use SwiftletSession for streaming chat, conversation caching, sampling controls, cancellation, and memory-pressure handling.
- CLI: `swiftlet chat` and `swiftlet generate` for local use and benchmarking, plus `swiftlet-repack` to build containers from MLX checkpoints or download them from Hugging Face or a mirror.
- Server: `swiftlet-server` exposes the OpenAI chat-completions API on loopback.
- App: Priv AI on iOS embeds SwiftletCore, and its source is published separately.

## Current Status and Tradeoffs

The README describes the runtime as working end to end with validated output, with current focus on kernel speed. Prefill runs at roughly decode speed, so long system prompts are slow, and batched prefill is the top roadmap item. The author notes the models chat and write like large models but recall facts like small ones. Layers are validated against mlx-lm reference implementations.

## Features
- Streams Mixture-of-Experts weights from SSD on demand
- Runs 35B and 80B Qwen models in low RAM
- Runs on iPhone and Mac
- Metal GPU forward pass with runtime-compiled shaders
- Qpack container format and repacker for MLX checkpoints
- Resumable Hugging Face streaming installer
- OpenAI-compatible loopback server
- SwiftletSession chat API with streaming deltas and conversation caching
- Gated DeltaNet linear attention support
- LFU plus recency expert cache
- Validated against mlx-lm reference implementations

## Integrations
Hugging Face, MLX, Qwen models, OpenAI-compatible chat clients, Priv AI iOS app

## Platforms
MACOS, IOS, CLI, DEVELOPER_SDK, API

## Pricing
Open Source

## Links
- Website: https://github.com/leonickson1/Swiftlet
- Documentation: https://github.com/leonickson1/Swiftlet
- Repository: https://github.com/leonickson1/Swiftlet
- EveryDev.ai: https://www.everydev.ai/tools/swiftlet