LittleLearner

LittleLearner is an academic research project from MPI for Intelligent Systems, ELLIS Institute Tübingen, and ETH Zürich that trains language models from scratch on a strictly filtered K–5 curriculum corpus. The project is led by Fanfei Li, Jana Zeller, Manuel Prada-Corral, and collaborators, and is accompanied by a paper on arXiv (2608.13545). A live 5B-parameter chat demo is hosted in-browser for direct experimentation.

What It Is

LittleLearner is a controlled sandbox for studying how language models acquire knowledge. Modern LMs are typically trained on everything at once, making it difficult to determine whether a capability was genuinely learned or merely elicited from latent pretraining data. LittleLearner addresses this by constraining the training distribution itself: an 88-billion-token corpus called LittleCurriculum, filtered from FineWeb-Edu through a five-stage pipeline aligned with U.S. Common Core standards for grades K–5. Concepts, facts, and vocabulary taught above Grade 5 are explicitly excluded. Three model scales (0.6B, 1.3B, and 5B parameters) are trained from scratch on this corpus, each paired with a matched "Unfiltered" control sharing the same architecture, token count, and training recipe.

Architecture and Model Variants

Each scale ships in three variants:

• Base: the raw pretrained checkpoint
• GRPO: a math specialist post-trained on MathCAMPS using reinforcement learning; responses may exhibit a tendency toward math-oriented output
• Chatty: variants tuned for general conversational behavior

All checkpoints are publicly released on Hugging Face under the littlelearner organization, making them directly downloadable for research use.

Key Research Findings

The project's central finding is that the pretraining filter sets the effective capability ceiling. In controlled experiments across three intervention types:

• Scaling model size improves in-scope (K–5) performance and extends modestly along the same learning trajectory, but yields little improvement on problems requiring capabilities outside the curriculum
• Post-training via GRPO significantly boosts in-scope K–5 capabilities but fails to recover out-of-scope (beyond-K–5) performance, even when post-training uses out-of-scope data
• In-context learning with tested prompting strategies does not unlock new reasoning capabilities beyond K–5 for the 5B model

These results support the interpretation that scaling, SFT+GRPO, and in-context learning amplify what the curriculum taught rather than enabling genuine out-of-scope acquisition.

Research Directions Enabled

Because LittleLearner's training exposure is explicitly specified, the project opens several tractable experimental directions:

• RL and discovery: capabilities emerging under RL can be attributed to the RL process itself, since the prior is restricted to K–5
• Continual learning: introducing new concepts (e.g., negative numbers) allows measurement of sample efficiency, retention, and interference near the knowledge boundary
• Educational science: specified exposure enables controlled human-model comparison to study whether models and children need similar exposure to learn concepts like fractions or make similar errors on word problems

Audience and Access

LittleLearner is explicitly a research artifact and is not designed or safety-aligned for children. It targets NLP researchers, cognitive scientists, and educators interested in mechanistic understanding of language model learning. The dataset (LittleCurriculum) is available on Hugging Face, all model checkpoints are publicly released, and a live chat interface runs the 5B chatty model directly in the browser for interactive exploration.