WASTE — Weight-Aware Streaming Tensor Engine

WASTE (Weight-Aware Streaming Tensor Engine) is an open-source, embeddable inference engine written in C that enables running enormous mixture-of-experts models on consumer hardware by streaming activated weights directly from fast NVMe storage rather than requiring all parameters to fit in RAM. It is developed by SQLite Cloud, Inc. and released under the Apache 2.0 license. The project's stated goal is to run the full 2.78-trillion-parameter Kimi K3 model on a 64 GB MacBook Pro, achieving approximately 0.6 tokens per second as measured on an M5 Pro machine.

What It Is

WASTE is a local inference engine targeting mixture-of-experts (MoE) models, where only a small fraction of parameters are active per token. Rather than loading the entire model into RAM, WASTE keeps the shared model trunk in memory and streams only the selected expert weights from disk on demand. The remaining RAM acts as a bounded expert cache. This architecture allows a 982 GB converted Kimi K3 container — derived from 1.42 TB of published weights — to run on a machine with as little as 29.06 GB of RAM (though 64 GB is the practical recommended minimum).

How the Streaming Architecture Works

The engine's design centers on several interlocking techniques:

• Expert streaming: The container format aligns each expert to a single read operation, so only the experts activated for each token are fetched from disk.
• Lookahead router: A predictive router starts reading the experts likely needed by the next layer while the current layer computes, overlapping I/O with computation without changing the model's output.
• Bounded expert cache: Unused RAM is used as a cache for recently accessed experts, reducing repeated disk reads. The README documents that shrinking the cache from 17.32 GB to 3.32 GB costs roughly 10% of throughput.
• Quantization: Experts use 3-bit residual vector quantization; more sensitive shared weights remain at 4 or 8 bits.
• Compressed KV cache: Kimi K3's linear attention and compressed latent KV cache reduce the KV footprint to approximately 0.21 GB at 4K context, compared to 11.25 GB for a standard attention model.

Storage bandwidth is the primary constraint. The README notes that a cold K3 token reads about 17 GB of experts, and the internal SSD on the test machine sustains 12.78 GB/s, while a tested USB enclosure managed only 0.94 GB/s.

Performance and Hardware Requirements

Measured on a 64 GB MacBook Pro with an M5 Pro and the model container on internal SSD:

The README explicitly notes that a 32 GB machine can open K3 but will page heavily. For users who want to try the engine without the full K3 requirements, Kimi-Linear (19 GB container, 1.28 GB RAM) is the recommended starting point. All layers are validated against a PyTorch reference; final logits agree within 3.6e-06.

Capabilities: Vision, Serving, and Embedding

Beyond text generation, WASTE supports multimodal inference. Kimi K3 is multimodal, and the engine accepts one or more images alongside text prompts via the --image flag or /image command in interactive mode. An optional Python server implements the OpenAI chat-completions API, supporting streaming, tool calls, structured output, thinking controls, and images. WASTE is also an embeddable C library (libwaste.a), with the public API exposed in src/waste.h. The inference path depends only on libc and pthreads — no BLAS, Python, CUDA, or other external dependency is required for CPU inference.

Update: v0.6.3

The latest release is v0.6.3, published on 2026-08-03. The repository was created on 2026-07-28 and has been updated rapidly since. The README describes the project as "currently changing very quickly," with a large QA run executed before each release. The format and API are explicitly noted as not yet frozen. Current research directions include CUDA, Metal, and other hardware-specific backends, which the README says "remain to be explored and may provide significant gains." The project is human-directed — ideas, hypotheses, and decisions are human-driven, while code is written by LLMs, a workflow the README describes as necessary for iterating on new algorithms at this scale.