# WASTE — Weight-Aware Streaming Tensor Engine

> An embeddable, dependency-free C inference engine that runs massive frontier models like Kimi K3 (2.78T parameters) on consumer hardware by streaming expert weights from NVMe storage.

WASTE (Weight-Aware Streaming Tensor Engine) is an open-source, embeddable inference engine written in C that enables running enormous mixture-of-experts models on consumer hardware by streaming activated weights directly from fast NVMe storage rather than requiring all parameters to fit in RAM. It is developed by SQLite Cloud, Inc. and released under the Apache 2.0 license. The project's stated goal is to run the full 2.78-trillion-parameter Kimi K3 model on a 64 GB MacBook Pro, achieving approximately 0.6 tokens per second as measured on an M5 Pro machine.

## What It Is

WASTE is a local inference engine targeting mixture-of-experts (MoE) models, where only a small fraction of parameters are active per token. Rather than loading the entire model into RAM, WASTE keeps the shared model trunk in memory and streams only the selected expert weights from disk on demand. The remaining RAM acts as a bounded expert cache. This architecture allows a 982 GB converted Kimi K3 container — derived from 1.42 TB of published weights — to run on a machine with as little as 29.06 GB of RAM (though 64 GB is the practical recommended minimum).

## How the Streaming Architecture Works

The engine's design centers on several interlocking techniques:

- **Expert streaming**: The container format aligns each expert to a single read operation, so only the experts activated for each token are fetched from disk.
- **Lookahead router**: A predictive router starts reading the experts likely needed by the next layer while the current layer computes, overlapping I/O with computation without changing the model's output.
- **Bounded expert cache**: Unused RAM is used as a cache for recently accessed experts, reducing repeated disk reads. The README documents that shrinking the cache from 17.32 GB to 3.32 GB costs roughly 10% of throughput.
- **Quantization**: Experts use 3-bit residual vector quantization; more sensitive shared weights remain at 4 or 8 bits.
- **Compressed KV cache**: Kimi K3's linear attention and compressed latent KV cache reduce the KV footprint to approximately 0.21 GB at 4K context, compared to 11.25 GB for a standard attention model.

Storage bandwidth is the primary constraint. The README notes that a cold K3 token reads about 17 GB of experts, and the internal SSD on the test machine sustains 12.78 GB/s, while a tested USB enclosure managed only 0.94 GB/s.

## Performance and Hardware Requirements

Measured on a 64 GB MacBook Pro with an M5 Pro and the model container on internal SSD:

| Model | Container | Minimum RAM | Decode speed |
|---|---|---|---|
| Kimi K3 2.78T | 982 GB | 29.06 GB | 0.45–0.62 tok/s |
| Kimi-Linear 48B | 19 GB | 1.28 GB | 10.65 tok/s |

The README explicitly notes that a 32 GB machine can open K3 but will page heavily. For users who want to try the engine without the full K3 requirements, Kimi-Linear (19 GB container, 1.28 GB RAM) is the recommended starting point. All layers are validated against a PyTorch reference; final logits agree within 3.6e-06.

## Capabilities: Vision, Serving, and Embedding

Beyond text generation, WASTE supports multimodal inference. Kimi K3 is multimodal, and the engine accepts one or more images alongside text prompts via the `--image` flag or `/image` command in interactive mode. An optional Python server implements the OpenAI chat-completions API, supporting streaming, tool calls, structured output, thinking controls, and images. WASTE is also an embeddable C library (`libwaste.a`), with the public API exposed in `src/waste.h`. The inference path depends only on libc and pthreads — no BLAS, Python, CUDA, or other external dependency is required for CPU inference.

## Update: v0.6.3

The latest release is v0.6.3, published on 2026-08-03. The repository was created on 2026-07-28 and has been updated rapidly since. The README describes the project as "currently changing very quickly," with a large QA run executed before each release. The format and API are explicitly noted as not yet frozen. Current research directions include CUDA, Metal, and other hardware-specific backends, which the README says "remain to be explored and may provide significant gains." The project is human-directed — ideas, hypotheses, and decisions are human-driven, while code is written by LLMs, a workflow the README describes as necessary for iterating on new algorithms at this scale.

## Features
- Stream expert weights from NVMe storage to run models beyond available RAM
- Lookahead router for overlapping disk I/O with computation
- Bounded expert cache using remaining RAM
- 3-bit residual vector quantization for experts
- Multimodal inference with image support
- OpenAI-compatible chat completions server
- Embeddable C library (libwaste.a) with public API
- No external dependencies for CPU inference (libc and pthreads only)
- Interactive chat mode with session persistence
- Supports macOS, Linux, and Windows (MinGW-w64)
- Validation against PyTorch reference for correctness
- Kimi-Linear 48B support for lower-resource environments

## Integrations
Kimi K3 (2.78T parameter model), Kimi-Linear 48B, OpenAI chat completions API (server compatibility), PyTorch (for model conversion and validation only), safetensors (for model conversion only)

## Platforms
WINDOWS, MACOS, LINUX, WEB, API, CLI

## Pricing
Open Source

## Version
v0.6.3

## Links
- Website: https://sqlite.ai
- Documentation: https://docs.sqlitecloud.io/
- Repository: https://github.com/sqliteai/waste
- EveryDev.ai: https://www.everydev.ai/tools/waste-inference-engine