TinyTurk Wiki - B_sweet (B (sweet spot))
Balanced - best structural diagnostics
This model is part of the TinyTurk research project on architectural design for tiny-scale Turkish language modeling. See companion models:
Research Repository
This model is part of the TinyTurk research project. Full source code, training scripts, ablation studies, and evaluation tools are available at:
https://github.com/stunmuffin/TinyTurk
Model Summary
| Property | Value |
|---|---|
| Architecture | Decoder-only Transformer (GPT-style, Pre-LN) |
| Parameters | 2,761,344 |
| Layers | 2 |
| Model dimension | 384 |
| Attention heads | 8 |
| FFN hidden dim | 512 (main) + 256 (micro) |
| Context length | 384 tokens |
| Vocabulary | 1024 BPE (SentencePiece) |
| Positional encoding | RoPE (theta = 10000) |
| Normalization | RMSNorm (eps = 1e-6) |
| Activation | GELU |
| Dropout | 0.10 |
| Weight tying | Yes |
| Language | Turkish |
Training
Trained from scratch on a 49,000-article subset of Turkish Wikipedia (barandinho/wikipedia_tr), tokenized with a custom 1024-vocab BPE tokenizer (UNK rate 0.078%).
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW (fused) |
| Learning rate | 3e-4 |
| LR schedule | Cosine with 5 percent warmup |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |
| Batch size | 16 |
| Epochs | 30 |
| Seed | 42 |
Evaluation
In-Domain (Wikipedia validation, 945 samples)
| Metric | Value |
|---|---|
| Validation loss | 2.3767 |
| Perplexity | 10.77 |
| Bits-per-char | 1.4653 |
Out-of-Domain (Turkish structural diagnostics)
See the GitHub repository for full results. Key observation: structural diagnostics peak at the 2.76M (B) configuration and degrade at larger scales, revealing a task-dependent optimum.
Architecture Notes
This model family tests three hypotheses departing from standard transformer design:
- Shallowness dominates depth - 2 layers outperform 3, 4, 6 at matched budget.
- Compressed FFN ratios - FFN = 1.3-2.2x d_model, not 4x.
- Width scales monotonically - with diminishing returns, following PPL proportional to N^-0.23.
Usage
Install:
pip install torch sentencepiece huggingface_hub
Python:
import torch
import sentencepiece as spm
from huggingface_hub import hf_hub_download
from model import TinyTurkGPTV27, TinyTurkV27Config
REPO = "stunmuffin/TinyTurk-Wiki-B-Sweet"
ckpt_path = hf_hub_download(REPO, "pytorch_model.bin")
tok_path = hf_hub_download(REPO, "tokenizer_wiki_bpe_1024.model")
ckpt = torch.load(ckpt_path, map_location="cpu")
config = TinyTurkV27Config(**ckpt["config"])
model = TinyTurkGPTV27(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()
sp = spm.SentencePieceProcessor(model_file=tok_path)
prompt = "Türkiye, "
ids = [sp.bos_id()] + sp.encode(prompt, out_type=int)
input_ids = torch.tensor([ids])
with torch.no_grad():
out = model.generate(input_ids, max_new_tokens=200,
temperature=0.7, top_k=40)
print(sp.decode(out[0].tolist()))
See example.py for a runnable script.
Limitations
- Wikipedia domain only. Performance on narrative, dialogue, or code is degraded.
- Small vocabulary (1024 BPE) - rare words may be split.
- Context 384 tokens. Not suitable for long documents.
- Raw pretrained LM. No instruction tuning.
- No safety alignment.
Citation
@misc{tinyturk_b_sweet_2026,
title = {TinyTurk Wiki - B_sweet: Turkish 2.76M parameter LM},
author = {TinyTurk Project},
year = {2026},
url = {https://huggingface.co/stunmuffin/TinyTurk-Wiki-B-Sweet}
}
TinyTurk is an experimental research artifact. Not a product.
- Downloads last month
- 476