# MiniMind

> An open-source project to train a 64M-parameter LLM from scratch in 2 hours on a single GPU, covering the full pipeline from tokenizer to RLAIF.

MiniMind is an open-source educational project by Jingyao Gong that lets anyone train a small but complete large language model entirely from scratch. The current flagship model, MiniMind-3, has 64M parameters and can be trained on a single NVIDIA RTX 3090 in roughly two hours. All core algorithms are implemented in native PyTorch with no high-level framework abstractions, making every line of code readable and understandable.

## What It Is

MiniMind is a full-stack LLM training framework and educational resource, not just a pretrained model. It covers the entire pipeline a production LLM goes through: custom BPE tokenizer training, pretraining on text-to-text data, supervised fine-tuning (SFT), LoRA adaptation, direct preference optimization (DPO), and reinforcement learning from AI feedback (RLAIF) via PPO, GRPO, and CISPO. The project also includes Agentic RL with multi-turn Tool-Use, adaptive thinking via `<think>` tags, and an OpenAI-compatible API server. Two model variants are available: a Dense 64M model and a MoE 198M/A64M model whose architecture is aligned with Qwen3/Qwen3-MoE.

## Architecture and Training Pipeline

The MiniMind-3 Dense model uses a Transformer Decoder-Only architecture with Pre-Norm + RMSNorm, SwiGLU activations, RoPE positional encoding with YaRN length extrapolation support, and grouped-query attention (8 query heads, 4 KV heads). The MoE variant extends this with 4 experts and top-1 routing. The training pipeline is staged:

- **Pretrain**: next-token prediction on `pretrain_t2t_mini.jsonl` or the full `pretrain_t2t.jsonl`
- **SFT**: multi-turn dialogue fine-tuning with Tool Call and reasoning data already mixed in
- **LoRA**: parameter-efficient fine-tuning implemented from scratch without `peft`
- **DPO**: preference optimization implemented natively in PyTorch
- **RLAIF (PPO/GRPO/CISPO)**: online RL with a reward model (e.g., InternLM2-1.8B-Reward) or rule-based signals
- **Agentic RL**: multi-turn Tool-Use rollouts with delayed reward, supporting SGLang as a decoupled rollout engine

## Compatibility and Deployment

MiniMind-3 is compatible with the broader open-source inference ecosystem. Pretrained weights are published in both native PyTorch `.pth` format and HuggingFace Transformers format on ModelScope and HuggingFace. The project supports:

- **vLLM** and **SGLang** for high-throughput serving
- **llama.cpp** and **ollama** for local CPU/GPU inference (including `ollama run jingyaogong/minimind-3`)
- **Streamlit** WebUI for interactive chat with thinking display and multi-turn Tool Call
- **OpenAI-compatible API** (`serve_openai_api.py`) for drop-in use with FastGPT, Open-WebUI, and Dify
- Single-card and multi-card (DDP, DeepSpeed) training with checkpoint resume

## Update: MiniMind-3 (April 2026)

The latest release, dated 2026-04-01, is a major update that introduces MiniMind-3 and MiniMind-3-MoE. Key changes include:

- Architecture aligned with Qwen3/Qwen3-MoE; shared expert design removed from MoE
- Tokenizer rebuilt with BPE + ByteLevel, adding tool call and thinking tokens
- New native Agentic RL training script (`train_agent.py`) supporting GRPO/CISPO in multi-turn Tool-Use scenarios
- RLAIF/Agentic RL rollout engine decoupled for flexible inference backends
- `serve_openai_api.py` and `web_demo.py` updated with `reasoning_content`, `tool_calls`, and `open_thinking` support
- LoRA weight merge and export via `scripts/convert_model.py`
- Standalone `train_reason.py` removed; thinking capability unified under `chat_template` + `open_thinking` toggle

The GitHub repository shows over 57,000 stars and active commits through mid-2026, with the latest release tagged as v2 ("MiniMind Docs") published in October 2025.

## Why It Matters for Learners

MiniMind is explicitly designed as a learning resource for people who want to understand LLMs by building one, not just fine-tuning one. The project avoids high-level abstractions from `transformers`, `trl`, and `peft`, implementing PPO, GRPO, CISPO, DPO, LoRA, and knowledge distillation from scratch in PyTorch. Evaluation support covers C-Eval, C-MMLU, ARC-Easy, PIQA, OpenBookQA, HellaSwag, and Social-IQa via the `lm-evaluation-harness` framework. The project has been cited in academic papers and a textbook published by Tsinghua University Press, according to the repository's acknowledgements section.

## Features
- Train a 64M LLM from scratch in ~2 hours on a single RTX 3090
- Full pipeline: Tokenizer → Pretrain → SFT → LoRA → DPO → PPO/GRPO/CISPO → Agentic RL
- Native PyTorch implementation with no high-level framework abstractions
- Dense (64M) and MoE (198M/A64M) model variants aligned with Qwen3/Qwen3-MoE
- OpenAI-compatible API server with tool_calls, reasoning_content, and open_thinking support
- Adaptive Thinking via <think> tags and open_thinking toggle
- Multi-turn Tool-Use Agentic RL with GRPO/CISPO
- YaRN RoPE length extrapolation for long-context inference
- Compatible with vLLM, ollama, llama.cpp, SGLang, and Transformers
- Streamlit WebUI with thinking display and multi-turn Tool Call
- Single-card and multi-card (DDP, DeepSpeed) training with checkpoint resume
- Evaluation on C-Eval, C-MMLU, ARC-Easy, PIQA, OpenBookQA, HellaSwag, Social-IQa
- LoRA weight merge and export via convert_model.py
- Knowledge distillation (black-box and white-box) implemented from scratch
- SwanLab and WandB integration for training visualization

## Integrations
HuggingFace Transformers, ModelScope, vLLM, ollama, llama.cpp, SGLang, FastGPT, Open-WebUI, Dify, Streamlit, WandB, SwanLab, DeepSpeed, lm-evaluation-harness, MNN, InternLM2-1.8B-Reward

## Platforms
MACOS, WEB, API, CLI

## Pricing
Open Source

## Version
minimind-3 (2026-04-01)

## Links
- Website: https://jingyaogong.github.io/minimind
- Documentation: https://github.com/jingyaogong/minimind/blob/master/README_en.md
- Repository: https://github.com/jingyaogong/minimind
- EveryDev.ai: https://www.everydev.ai/tools/minimind