open-athena/Snowflake-d512-Datakit-SFT-262K-2026.10.22

Snowflake d512 Datakit SFT 262K — 2026.10.22

Research artifact. This model has not been properly tested or evaluated and is not necessarily secure. Use at your own risk in production settings.

Snowflakes are smaller Grug mixture-of-experts models that apply the Snowball 2026.10.05 recipe to smaller Snowball-family checkpoints. This release contains the width-512 model after 64K long-context extension and Datakit midtraining, 262K extension, and 3,207 supervised fine-tuning updates. The release date in the name is October 22, 2026. The configured 262,144-token context has not been validated by long-context generation tests.

Training data

Pretraining scheduled 94,499,766,272 positions. Each posttraining stage preserves the corresponding fraction of Snowball's 8,959,033,344,000-position pretraining budget, rounded to whole updates. Each update schedules 1,048,576 positions. These counts include padding in packed sequences and do not measure distinct text tokens.

Stage	Context	Packed sequences/update	Updates	Scheduled positions	Peak base LR
lce-64k	65,536	16	10,126	10,617,880,576	0.000789724018
lce-262k	262,144	4	675	707,788,800	0.000253633991
sft	262,144	4	3,207	3,362,783,232	4.80367408e-05

The 64K stage uses Datakit phase-2 data. The 262K stage uses long-context pretraining data that repeats stores of documents longer than 64K eight times for each copy of the shorter-document store within each domain/quality bucket. SFT targets 79.6543% supervised data and 20.3457% pretraining replay: approximately 2,678,600,648 SFT positions and 684,182,584 replay positions out of 3,362,783,232 scheduled positions. Its 216 underlying SFT sources use the full 10.05 source allocation weights. Small sources are pooled before sampling. The smaller run samples a proportional budget; it does not force the full-size model's approximately one-epoch schedule. Integer sampling and pooling affect realized source counts.

Source family	Nominal share of complete SFT run
nemotron_sft_v3	64.2349%
open_swe_traces	10.8340%
agenttrove	1.6438%
openthoughts4-code-glm-5.2-n4	1.3809%
penfever-traces	1.3556%
ultrachat-persona-conversations	0.1024%
agenttrove-glm53-compactions	0.0624%
identity-data	0.0249%
science-tool-use-conversations	0.0066%
glm-5.2-kernelgym-rollouts	0.0053%
wildchat-glm53-format-completions	0.0030%
synthetic-misconceptions-conversations	0.0006%
Long-context pretraining replay	20.3457%

training_data_mix.json records the source allocation weights and nominal budgets. pretraining_replay_mix.json records replay weights and stores. training_schedule.json records stage budgets, learning rates, source checkpoints, and source-file hashes.

Model and provenance

Property	Value
Architecture	GrugMoeForCausalLM (grug_moe)
Total parameters	746,607,104
Experts	256 routed experts, top-4 routing, plus a shared expert
Layers / hidden size	6 / 512
Attention heads / KV heads	4 / 1
Expert / shared expert intermediate size	256 / 512
Vocabulary size	128,256
Configured context length	262,144 tokens
Sliding window	2,048 tokens
QK multiplier	1.75
Final checkpoint step	104,130
SFT updates / starting step	3,207 / 100,923
Export	BF16 safetensors
Training hardware	TPU v4-32 in us-central2-b

The model uses full causal attention every fourth layer and in the last layer; other layers use the sliding window. Long attention layers omit rotary position embeddings and partial key offsets, matching the source checkpoint family.

The full optimizer state is resumed between stages. SFT begins without a new warmup. The peak SFT base learning rate is 4.80367408e-05, decaying to 10% of peak during the final 10% of updates. Learning rates use the scaled May Recipe rule documented in training_schedule.json; they were not separately tuned for this ladder.

Updates to seven special-token rows in both input embeddings and the LM head are multiplied by √32. Router weights remain trainable; router biases and their deferred load-balancing updates remain frozen during SFT. The pending router-bias update carried from before SFT is applied when exporting the final checkpoint.

Seven LM-head rows are initialized from single-token anchors: 128006 from 3560 ( role), 128007 from 1984 ( message), 128002 from 1781 ( think), 128003 from 4320 ( answer), 128005 from 5507 ( tool), 128011 from 842 ( end), and 128009 from EOS (128001). Only those LM-head optimizer moments are reset. Input embeddings and their optimizer moments are preserved at initialization and remain trainable. BOS and EOS are preserved; no new tokens are added.

• SFT metrics
• Training library base
• Ladder posttraining source snapshot, with individual file hashes in training_schedule.json
• Native final checkpoint: gs://marin-us-central2/users/held/snowball-overtrained/d512/sft/2026.10.19.2/checkpoints/step-104130
• Other completed Snowflake model

Chat and generation settings

The tokenizer and templates are pinned to the full 10.05 release. Omitting enable_thinking defaults to thinking. For a vLLM chat request, select nonthinking mode with:

{"chat_template_kwargs": {"enable_thinking": false}}

For local prompt formatting:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("open-athena/Snowflake-d512-Datakit-SFT-262K-2026.10.22")
prompt_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 17 times 23?"}],
    tokenize=True,
    return_dict=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

Generation stops on both <|end_of_text|> (128001) and <|eot_id|> (128009). The tokenizer's EOS remains 128001. Preserve both stop IDs in other runtimes. Thinking boundaries are <|start_think|> (128002) and <|end_think|> (128003); decode with skip_special_tokens=False to retain them. Tools use JSON inside <tool_call>...</tool_call>; arguments may be objects or JSON strings. Tool-call delimiters remain ordinary text. The inference template adds the thinking default to the training template.

Serving and evaluation

These HF-format weights require a runtime with Grug MoE support. Stock Transformers and stock vLLM do not implement this architecture. The downloadable HF safetensors can be loaded through Levanter's SnowballConfig and SnowballLMHeadModel; the Marin GPU vLLM fork contains a Grug implementation with normalized sigmoid routing. GPU vLLM load and decode passed on one NVIDIA H100 using the pinned Marin fork, BF16 weights, eager execution, a 1,024-token runtime limit, and a 1 GiB KV cache. The next-token probe returned the same token ID as the native reference. Each server was launched with a pinned --revision, a distinct port, and a served-model name containing the width and HF revision. Each response's model field matched that configured name. vllm_validation.json records the pinned HF revision, runtime, probe, and four generated prompt responses. These checks do not validate 262K serving, throughput, or general model quality.

Every exported tensor was reloaded and checked for exact equality against the native checkpoint weights after applying its deferred router-bias update and BF16 conversion. The downloadable weights include that update. A 128-position native/Levanter HF forward probe also passed in FP32 using those same BF16 weights cast to FP32. export_complete.json records numerical results. These checks do not establish generation quality, safety, or performance at the configured context length. No held-out capability or long-context benchmark evaluation has been run for this release. The four inference prompts are qualitative examples, not a capability benchmark.

The tested runtime settings can be reproduced with the Marin GPU fork installed:

vllm serve open-athena/Snowflake-d512-Datakit-SFT-262K-2026.10.22 \
    --runner generate --tensor-parallel-size 1 --dtype bfloat16 \
    --max-model-len 1024 --max-num-batched-tokens 1024 --max-num-seqs 1 \
    --kv-cache-memory-bytes 1073741824 --enforce-eager --port 8512

Choose a distinct port for each concurrent server. For reproducible tests, pin --revision to the HF revision recorded in vllm_validation.json.

License

The model materials are released under the OpenMDW License Agreement, version 1.1.