LittleLearner

Research checkpoints for LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure — a study of how filtering the pretraining corpus to K–5 (kindergarten through grade 5) educational content affects a language model's downstream capabilities.

This organisation hosts LittleCurriculum, the K–5-filtered pretraining dataset, as well as the paired LittleLearner (K–5-filtered) and Unfiltered (baseline general corpus) checkpoints at three sizes, plus matched post-trained variants. All checkpoints share the same architecture and tokenizer, so a LittleLearner X B and Unfiltered X B differ only in what data they saw during pretraining — the controlled comparison the paper is built around.

📄 Paper: LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure · ✉️ Contact: fanfei.li@tuebingen.mpg.de

Naming conventions

Two model families:

• LittleLearner — pretrained on the K–5-filtered corpus.
• Unfiltered — pretrained on the same corpus without the K–5 filter (baseline general corpus).

Important inference notes

All chatty checkpoints share two non-standard defaults. Please read the individual model cards for the exact runnable snippet, but at a glance:

1. No system prompt. These models were SFT'd without a system message; leaving transformers' default system prompt on, or adding your own, measurably degrades on-task accuracy. Build the conversation as messages = [{"role": "user", "content": ...}] only — no system role.
2. Custom stop tokens. vLLM's default stop-token list derived from the tokenizer is not sufficient — generation can run past the assistant turn. Pass the stop_token_ids=[…] documented on each model card.

Base checkpoints do not carry a chat template and should be used in completion mode.

Citation

@article{littlelearner2026,
  title   = {LittleLearner: Language Models Under Pedagogically-Controlled Knowledge Exposure},
  author  = {Fanfei Li and Jana Zeller and Manuel Prada-Corral and Thadd{\"a}us Wiedemer
             and Prasanna Mayilvahanan and Ryan Cotterell and Wieland Brendel},
  journal = {arXiv preprint arXiv:26xx.xxxxx},
  year    = {2026}
}