Add TinyBrainBot-360M-A75M-v4.1-MoE-Base and TinyBrainBot-350M-V3-Base
Adds two more TinyBrainBot models (org entry already exists), both trained from scratch, open weights, no custom code.
TinyBrainBot-360M-A75M-v4.1-MoE-Base - a mixture-of-experts model: 383,429,376 parameters in total (358.9M non-embedding + 32k-vocab tied embedding), of which 75.7M non-embedding are active per token (24 experts per layer, top-4 routing). ~18.2B pretraining tokens, 2,048 context. Loads as a standard Transformers MixtralForCausalLM. After pretraining, 4 experts per layer were fine-tuned in place on factual Q&A text with every other weight frozen - one model, not a merge.
| Benchmark | Score |
|---|---|
| ARC-Easy (acc_norm) | 55.64 |
| ARC-Challenge (acc_norm) | 32.34 |
| HellaSwag (acc_norm) | 37.27 |
| PIQA (acc_norm) | 66.32 |
| ArithMark-3 (length-normalized) | 42.00 |
TinyBrainBot-350M-V3-Base - dense, 348,342,912 parameters (311.5M non-embedding + 32k-vocab tied embedding), ~22.7B tokens, final training checkpoint (not a merge). Loads as a standard LlamaForCausalLM.
| Benchmark | Score |
|---|---|
| ARC-Easy (acc_norm) | 56.61 |
| ARC-Challenge (acc_norm) | 33.28 |
| HellaSwag (acc_norm) | 35.90 |
| PIQA (acc_norm) | 66.97 |
| ArithMark-3 (length-normalized) | 39.30 |
Zero-shot, full test sets, lm-eval 0.4.13 (float32); ArithMark-3 with the official bencharithmark-3.py (float32).
To reproduce:
lm_eval --model hf --model_args pretrained=nkthebass/tinybrainbot-360m-a75m-v4.1-moe-base,dtype=float32 --tasks arc_easy,arc_challenge,hellaswag,piqa --num_fewshot 0
python bencharithmark-3.py --model nkthebass/tinybrainbot-360m-a75m-v4.1-moe-base --dtype float32
(same with nkthebass/tinybrainbot-350mV3-base)