samir-vllm
A small LLM inference engine in C++ and CUDA that teaches inference from first principles with video tutorials and study notes.
At a Glance
MIT-licensed engine source code; notes and Llama materials are excluded from MIT and carry separate terms.
Engagement
Available On
Listed Nov 2026
About samir-vllm
samir-vllm is an educational C++17/CUDA project by Samir Sarwar that builds a small LLM inference engine for Llama 3.2 1B Instruct. It pairs the source code with a video build series and handwritten notes covering what happens between a prompt and a generated token.
What It Is
The project is a learning-oriented inference engine that loads model weights, tokenizes prompts, and runs the transformer on an NVIDIA GPU. It walks through each stage of inference: SafeTensor weight loading, embedding gather, RMSNorm, RoPE, Q/K/V projections with cuBLAS, causal self-attention, the SwiGLU feed-forward network, decode with a KV cache, batching, and paged attention.
How the engine works
The engine reads model.safetensors directly and copies BF16 weights to GPU memory. A local BPE tokenizer handles Unicode, special tokens, and the Instruct chat format. Prefill runs all 16 layers, and a GPU argmax picks each next token using greedy generation. A shared paged KV cache stores 16-token pages with per-sequence block tables, and decode attention uses FP32 online softmax over cached pages.
Batching and scope
The command-line program accepts one or more prompts and a max_new_tokens value. It supports static batching and a continuous scheduler that admits queued requests according to page budgets, plus a --kv-cache-mib option and a --stats report. Model dimensions and the 2,048-token context are fixed. The README notes that greedy outputs may differ between individual and batched runs because of BF16 rounding.
Learning materials and licensing
The repository links to a video series of 6+ hours (Part 0 through a paged-attention finale), per-video study notes in Markdown and PDF, and an 80+ page handwritten notes PDF. The source code is MIT licensed, while the notes and handwritten materials are all rights reserved. Llama weights and tokenizer keep Meta's Llama 3.2 Community License.
Community Discussions
Be the first to start a conversation about samir-vllm
Share your experience with samir-vllm, ask questions, or help others learn from your insights.
Pricing
Open Source (MIT engine)
MIT-licensed engine source code; notes and Llama materials are excluded from MIT and carry separate terms.
- Engine source in src/, include/, tests/, python/ plus CMakeLists.txt, .gitignore and README under MIT License
- Notes (proper_notes/, notes.md, podLibraryInstall.md, handwritten notes, PDFs) are all rights reserved, not MIT; personal reading and study permitted
- Bundled Llama 3.2 tokenizer and Llama weights are under Meta's Llama 3.2 Community License, excluded from MIT
Capabilities
Key Features
- Direct SafeTensor weight loading to GPU
- Local Unicode BPE tokenizer with Instruct chat format
- CUDA kernels for embedding gather, RMSNorm and RoPE
- cuBLAS Q/K/V projections
- Dense causal prefill across 16 layers
- Paged KV cache with block tables
- Paged decode attention with FP32 online softmax
- Static and continuous batching
- Configurable KV cache size and stats reporting
- Video series and handwritten study notes
Integrations
Demo Video

