Tiny-LLM

Tiny-LLM is a free, open-source hands-on course created by Chi (Alex Chi Z.) and Connor, two systems engineers who wanted to understand LLM inference end to end without wading through hundreds of thousands of lines of CUDA-optimized code. The course is published as an interactive book at skyzh.github.io/tiny-llm and is licensed under Apache 2.0 (code) and CC BY-NC-SA 4.0 (book content). It is designed for engineers who already understand deep learning basics and want to build a working LLM serving stack from first principles.

What It Is

Tiny-LLM is a four-week, project-based course that guides learners through implementing a complete LLM inference and serving system using Apple's MLX framework on Apple Silicon. Rather than calling high-level neural-network layers, students implement each operator — attention, RoPE, GQA, RMSNorm, KV cache, quantized kernels, paged attention, and more — in Python, C++, or Metal. MLX serves as the correctness oracle and performance baseline throughout. The course is explicitly positioned as an LLM-serving counterpart to CMU's Needle project from the Deep Learning Systems course.

Four-Week Learning Path

The course is structured as a cumulative four-week sequence, where each week builds on the interfaces and mechanisms established by the previous one:

• Week 1 — From Matmul to Text: Build a Qwen3 model from raw mlx.core array operations, covering attention, RoPE, grouped-query attention, RMSNorm, MLP, sampling, and the autoregressive generation loop.
• Week 2 — A Step Closer to vLLM: Add a KV cache, establish a synchronized MLX baseline, then implement quantized decode matvec, fused model kernels, tiled prefill, and split-K scheduling.
• Week 3 — Build a Mini vLLM: Introduce continuous batching, chunked prefill, paged KV cache, direct paged attention, and paged FlashAttention. Optional chapters cover speculative decoding and Mixture of Experts.
• Week 4 — Build a Coding Agent: Construct a bounded, validated agent loop connected to a local workspace, with sessions, context compaction, inspect-and-steer pauses, deterministic evaluation, and branch-and-select capabilities.

Why MLX and Apple Silicon

The course authors chose MLX and Apple Silicon because the unified memory architecture gives learners direct access to Metal kernels and the complete inference path on a single consumer machine, without requiring an expensive NVIDIA GPU setup. Qwen3-4B is the primary target model — large enough to expose real weight-bandwidth, attention, and cache costs, but small enough to iterate on locally. The course provides a model-size guidance table for MacBooks with 16 GB, 32 GB, and 64 GB of unified memory.

Architecture and Implementation Design

The repository separates the learner implementation (tiny_llm package) from the reference solution (tiny_llm_ref), which is used by tests and the benchmark appendix. The course does not use high-level neural-network layers from MLX; when a chapter teaches an operator, the student implements it directly. Week 2 introduces explicit "operator off-ramps" — documented seams where a learner can substitute an MLX built-in (e.g., mx.fast.scaled_dot_product_attention) instead of writing a custom Metal kernel, while preserving the surrounding course interfaces. The roadmap diagram in the book tracks which weeks depend on which interfaces and which optimizations are optional.

Current Status and Roadmap

The GitHub repository shows active development as of mid-2026. All 28 chapters across Weeks 1–4 (including Days 1–9 of Week 4) have completed code, tests, and documentation. Week 4 chapters are still undergoing the authors' editorial audit pass. The repository has accumulated over 4,500 GitHub stars and 369 forks since its creation in April 2025, reflecting significant community interest. A Discord community is available for learners to study together. Topics not yet covered by the course include quantized or compressed KV caches, cross-request prefix caching, fine-tuning, and long-context techniques.