MiniMind

MiniMind is an open-source educational project by Jingyao Gong that lets anyone train a small but complete large language model entirely from scratch. The current flagship model, MiniMind-3, has 64M parameters and can be trained on a single NVIDIA RTX 3090 in roughly two hours. All core algorithms are implemented in native PyTorch with no high-level framework abstractions, making every line of code readable and understandable.

What It Is

MiniMind is a full-stack LLM training framework and educational resource, not just a pretrained model. It covers the entire pipeline a production LLM goes through: custom BPE tokenizer training, pretraining on text-to-text data, supervised fine-tuning (SFT), LoRA adaptation, direct preference optimization (DPO), and reinforcement learning from AI feedback (RLAIF) via PPO, GRPO, and CISPO. The project also includes Agentic RL with multi-turn Tool-Use, adaptive thinking via <think> tags, and an OpenAI-compatible API server. Two model variants are available: a Dense 64M model and a MoE 198M/A64M model whose architecture is aligned with Qwen3/Qwen3-MoE.

Architecture and Training Pipeline

The MiniMind-3 Dense model uses a Transformer Decoder-Only architecture with Pre-Norm + RMSNorm, SwiGLU activations, RoPE positional encoding with YaRN length extrapolation support, and grouped-query attention (8 query heads, 4 KV heads). The MoE variant extends this with 4 experts and top-1 routing. The training pipeline is staged:

• Pretrain: next-token prediction on pretrain_t2t_mini.jsonl or the full pretrain_t2t.jsonl
• SFT: multi-turn dialogue fine-tuning with Tool Call and reasoning data already mixed in
• LoRA: parameter-efficient fine-tuning implemented from scratch without peft
• DPO: preference optimization implemented natively in PyTorch
• RLAIF (PPO/GRPO/CISPO): online RL with a reward model (e.g., InternLM2-1.8B-Reward) or rule-based signals
• Agentic RL: multi-turn Tool-Use rollouts with delayed reward, supporting SGLang as a decoupled rollout engine

Compatibility and Deployment

MiniMind-3 is compatible with the broader open-source inference ecosystem. Pretrained weights are published in both native PyTorch .pth format and HuggingFace Transformers format on ModelScope and HuggingFace. The project supports:

• vLLM and SGLang for high-throughput serving
• llama.cpp and ollama for local CPU/GPU inference (including ollama run jingyaogong/minimind-3)
• Streamlit WebUI for interactive chat with thinking display and multi-turn Tool Call
• OpenAI-compatible API (serve_openai_api.py) for drop-in use with FastGPT, Open-WebUI, and Dify
• Single-card and multi-card (DDP, DeepSpeed) training with checkpoint resume

Update: MiniMind-3 (April 2026)

The latest release, dated 2026-04-01, is a major update that introduces MiniMind-3 and MiniMind-3-MoE. Key changes include:

• Architecture aligned with Qwen3/Qwen3-MoE; shared expert design removed from MoE
• Tokenizer rebuilt with BPE + ByteLevel, adding tool call and thinking tokens
• New native Agentic RL training script (train_agent.py) supporting GRPO/CISPO in multi-turn Tool-Use scenarios
• RLAIF/Agentic RL rollout engine decoupled for flexible inference backends
• serve_openai_api.py and web_demo.py updated with reasoning_content, tool_calls, and open_thinking support
• LoRA weight merge and export via scripts/convert_model.py
• Standalone train_reason.py removed; thinking capability unified under chat_template + open_thinking toggle

The GitHub repository shows over 57,000 stars and active commits through mid-2026, with the latest release tagged as v2 ("MiniMind Docs") published in October 2025.

Why It Matters for Learners

MiniMind is explicitly designed as a learning resource for people who want to understand LLMs by building one, not just fine-tuning one. The project avoids high-level abstractions from transformers, trl, and peft, implementing PPO, GRPO, CISPO, DPO, LoRA, and knowledge distillation from scratch in PyTorch. Evaluation support covers C-Eval, C-MMLU, ARC-Easy, PIQA, OpenBookQA, HellaSwag, and Social-IQa via the lm-evaluation-harness framework. The project has been cited in academic papers and a textbook published by Tsinghua University Press, according to the repository's acknowledgements section.