TinyBrainBot-360M-A75M-v4.1-MoE-Base

A mixture-of-experts language model trained from scratch on two Tesla V100s. Every layer holds 24 experts and each token uses 4 of them, so the model stores 359M parameters but runs only 75.7M per token (non-embedding; plus a 24.6M tied embedding, 383M in total). It was trained on 18.2B tokens with a 2,048-token context, then given one expert-tuned patch for factual question answering (below).

It is our strongest model on the Open SLM Leaderboard's settings, ahead of our dense 350M while computing about a quarter as much per token.

Model	Params	Train tokens	HellaSwag	ARC-Easy	ARC-Challenge	PIQA	ArithMark-3	Intelligence Index
TinyBrainBot-360M-A75M (this)	383M (75.7M active)	18.2B	37.27	55.64	32.34	66.32	42.00	24.40
(raw acc)			32.53	59.76	28.84	66.54
TinyBrainBot-350M-V3-Base (dense)	348M	22.7B	35.90	56.61	33.28	66.97	39.30	23.96
(raw acc)			31.64	62.25	30.38	65.78
TinyBrainBot-100M-V3-Base (dense)	100M	25B	32.62	54.67	30.12	66.21	37.40	20.97
(raw acc)			29.95	58.59	26.62	65.72
SmolLM2-135M	135M	2T	43.22	58.63	29.69	68.44	39.20	27.13
OPT-350M	331M	180B	36.82	40.19	23.72	64.36	38.30	17.88
GPT-2 Medium	355M	-	39.38	43.60	25.00	66.38	33.80	19.72
GPT-2	124M	-	31.26	39.35	22.35	62.08	35.70	13.58

Full test sets, zero-shot, acc_norm, EleutherAI lm-eval-harness, plus the official ArithMark-3 script. The Intelligence Index is the leaderboard's: the normalised average of HellaSwag, ARC, PIQA and 0.65x ArithMark-3. TinyBrainBot rows were run by us; the other rows are the leaderboard's published numbers.

Quick start

It is a standard Mixtral model and needs no custom code:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nkthebass/tinybrainbot-360m-a75m-v4.1-moe-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

ids = tok("Once upon a time, a little fox", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80)       # defaults from generation_config.json
print(tok.decode(out[0], skip_special_tokens=True))

GGUF (llama.cpp, LM Studio, Ollama): tinybrainbot-360m-a75m-v4.1-moe-base-Q8_0.gguf (409 MB) or -F16.gguf (768 MB). Both tokenize exactly like the transformers version and give the same greedy output.

llama-completion -m tinybrainbot-360m-a75m-v4.1-moe-base-Q8_0.gguf -no-cnv -p "Once upon a time, a little fox" -n 80 --temp 0.7 --repeat-penalty 1.15

This is a base model: it continues text, it does not follow chat instructions. Default sampling: temperature 0.7, top_p 0.9, repetition_penalty 1.15. A chat version is in progress.

Samples

Unedited, at the default settings.

Once upon a time, a little fox named Max lived in a small village. One day, Max heard his name mentioned by someone else. He asked around the town and found that everyone had lost their way in the forest. But then, he noticed something strange happening - some trees were open wide! All of the animals were moving quickly toward each other because they wanted to find food and shelter. The fox realized that the path wasn't clear at all; instead, it seemed like it

Question: Why do we have seasons on Earth? Answer: The tilt of Earth's axis causes changing sunlight angles.

The best way to learn a new language is by engaging with it. This means watching movies, reading books, attending cultural events, and practicing speaking the language regularly. It's not just about memorizing words; it's also about understanding grammar rules and vocabulary lists. By doing so, you will gain confidence in communicating effectively and expressing yourself in the target language. Another important aspect of learning a new language is improving your listening skills. Listening to native speakers can help you understand conversations and

All benchmarks

Benchmark (0-shot)	this model	before the expert patch
ARC-Easy	55.9	54.2
ARC-Easy (raw acc)	59.8	59.2
ARC-Challenge	32.3	31.9
ARC-Challenge (raw acc)	28.8	27.8
SciQ	73.3	72.5
SciQ (raw acc)	81.8	82.1
OpenBookQA	34.4	33.8
OpenBookQA (raw acc)	23.6	23.2
MMLU	24.2	24.4
HellaSwag	41.5	41.9
HellaSwag (raw acc)	35.8	35.4
PIQA	66.3	66.8
PIQA (raw acc)	66.5	66.6
WinoGrande	52.4	52.6
Social IQa	39.7	39.1
CommonsenseQA	21.6	20.9
LAMBADA (OpenAI)	28.6	28.5
BoolQ	57.8	58.6
MathQA	23.2	23.9
MathQA (raw acc)	24.1	24.7
Average (13)	42.4	42.2
LAMBADA perplexity (lower is better)	65.7	64.2

lm-eval-harness, 0-shot, up to 2,000 examples per task (HellaSwag and MMLU use their first 2,000; the leaderboard table above uses full sets). Each task's first row is acc_norm where the task reports it, acc otherwise; the "(raw acc)" row under it is acc. "Before the expert patch" is the same model at the end of pretraining.

Model

Architecture	Mixtral-style sparse MoE decoder: RoPE, RMSNorm, grouped-query attention, SwiGLU experts
Shape	hidden 768, 12 layers, 12 heads of 64, 4 KV heads; 24 experts of width 512 per layer, top-4 routing
Parameters	358.9M non-embedding total, 75.7M active per token, + 24.6M tied embedding (383.4M total)
Context	2,048 tokens
Vocabulary	32,000 (SentencePiece, byte fallback)

Routing is Mixtral's: softmax over the 24 experts, keep the top 4, renormalise their weights. A 0.01 load-balancing loss keeps every expert in use. The experts run on the sigma-MoE Triton kernel (Csordas et al.), which fuses token routing into the matrix multiply, so training ran at about 30k tokens/s on the two V100s.

Training

Pretraining: 16.7B tokens, fp16, AdamW (0.9, 0.95), weight decay 0.1, peak LR 8e-4, 262k tokens per step, warmup-stable-decay.

Group	Share	Sources
Web	35%	FineWeb-Edu 27, DCLM 9
Reference	14%	Wikipedia article leads 12, full Wikipedia 1.5
Math	11%	MegaMath-Web-Pro 5, FineMath-4+ 4, Cosmopedia auto-math 1, Orca-Math 1
Narrative	10%	Project Gutenberg 4, TinyStories 4, BookCorpusOpen 3
Code	8%	The Stack (smol) 7, Python/JS 1.5
Synthetic textbooks	8%	Cosmopedia (Stanford, OpenStax)
Q&A and facts	7%	distilled Q&A, short facts, cause-and-effect explanations, short reasoning
Educational PDFs	7%	FinePDFs-Edu

Anneal: 1.5B tokens while the learning rate decays to zero: mostly distilled Q&A and short facts, with explanations and short reasoning, Wikipedia, Cosmopedia (including WikiHow), FineMath, FinePDFs and short generated everyday scenes and stories.

Expert patch: factual Q&A. In each layer, the 4 experts that factual question-answer text uses most (relative to ordinary text) were trained for 131M tokens on plain-text Q&A, short facts and explanations, mixed with pretraining text. Everything else stayed frozen - attention, norms, embeddings, router and the other 20 experts per layer - and was checked bit-for-bit after every step. It is one model fine-tuned in place, not a merge. The patch raised ARC-Easy and ARC-Challenge without moving the other benchmarks (see "All benchmarks").

Limitations

• A small base model: fluent, but it states wrong facts with confidence, and it does not follow instructions.
• Not for code: the tokenizer collapses whitespace, so newlines and indentation are lost.
• English only.

Reproduce

lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-360m-a75m-v4.1-moe-base,dtype=float32 \
  --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 16

TinyBrainBot v4.1 (mixture of experts)

• TinyBrainBot-360M-A75M-v4.1-MoE-Base (this model)
• TinyBrainBot-61M-A10M-v4.1-MoE-Base, the first MoE pilot