stunmuffin/TinyTurk-v2.7d-bigstories-384

TinyTurk v2.7d — BigStories 384

TinyTurk v2.7d BigStories-384 is a 1.12M-parameter Turkish causal language model with a 384-token context window, fine-tuned from BigStories-256 on the Turkish TinyStories dataset. It produces longer, more fluent Turkish text than its 128-token and 256-token predecessors.

Model Summary

Property	Value
Architecture	Decoder-only Transformer (GPT-style, Pre-LN)
Parameters	1,121,920
Layers	4
Model dimension	128
Attention heads	4 (head_dim = 32)
FFN hidden dim	512 (main) + 256 (micro)
Context length	384 tokens
Vocabulary	512 BPE (SentencePiece)
Positional encoding	RoPE (theta = 10,000)
Normalization	RMSNorm (eps = 1e-6)
Activation	GELU
Dropout	0.10
Weight tying	Yes
Language	Turkish

Training Pipeline

This model is the third stage of a progressive context-length scaling:

1. BigStories — trained from scratch, context = 128, 15 epochs
2. BigStories-256 — fine-tuned, context = 256, 10 epochs
3. BigStories-384 — fine-tuned, context = 384, 10 epochs (this model)

Hyperparameter	Value
Optimizer	AdamW
Learning rate	8e-5 (fine-tune)
LR schedule	Cosine decay with 5 percent warmup
Weight decay	0.01
Gradient clipping	1.0
Batch size	12
Epochs	10
Seed	42
Base checkpoint	tiny_gpt_v2.7d_bigstories_256_best.pt

Training time: ~10 minutes on a single modern GPU.

Evaluation

Language Modeling

Metric	Value
Validation cross-entropy	1.3046
Validation perplexity	3.69

Note: PPL is computed on 384-token windows, which are harder than shorter windows. The 256-token model reports a lower PPL (3.60) but on an easier evaluation.

Progressive Context Scaling

Model	Context	Val PPL	Output Length
BigStories	128	3.99	100 tokens
BigStories-256	256	3.60	200 tokens
BigStories-384	384	3.69	300 tokens

Generation Samples

Prompt: "Bir zamanlar kucuk bir kedi vardi."

Bir zamanlar kucuk bir kedi vardi. Kedi her zaman cok fazla guzel cicek yemek istiyordu. Bir gun bir magazaya gitti. Cicekleri gordu ve onu kendisine istedi. Annesinden "Hadi bakalim!" diye bagirdi...

Observation: Sentences are fluent and grammatically correct. Character continuity is partially maintained. Narrative coherence is still limited.

Usage

Install:

pip install torch sentencepiece huggingface_hub

Python:

import torch
import sentencepiece as spm
from huggingface_hub import hf_hub_download
from model import TinyTurkGPTV27, TinyTurkV27Config

REPO = "stunmuffin/TinyTurk-v2.7d-bigstories-384"

ckpt_path = hf_hub_download(REPO, "pytorch_model.bin")
tok_path = hf_hub_download(REPO, "tokenizer_v11_bpe_512.model")

ckpt = torch.load(ckpt_path, map_location="cpu")
config = TinyTurkV27Config(**ckpt["config"])
model = TinyTurkGPTV27(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

sp = spm.SentencePieceProcessor(model_file=tok_path)

prompt = "Bir zamanlar kucuk bir kedi vardi."
ids = [sp.bos_id()] + sp.encode(prompt, out_type=int)
input_ids = torch.tensor([ids])

with torch.no_grad():
    out = model.generate(input_ids, max_new_tokens=300,
                         temperature=0.9, top_k=40)

print(sp.decode(out[0].tolist()))

See example.py for a runnable script.

Limitations

Important: This model has several known limitations:

1. Narrative coherence: The model produces fluent sentences but does not maintain a consistent story arc. Character continuity is partial.
2. Translated training data: The corpus is an automatic translation of TinyStories (English). English proper names (Alice, Tom, Tim) may appear in Turkish output. Some phrases are literal translations.
3. Small vocabulary: 512 BPE subwords.
4. Model size: 1.12M parameters is too small for robust narrative modeling.
5. No instruction tuning: Raw pretrained language model.
6. No safety alignment: Outputs may be incoherent or inappropriate.

Intended Use

• Research on context-length scaling in tiny models
• Turkish sentence-level and short-paragraph generation
• Educational demonstrations
• On-device inference experiments

Out-of-Scope Use

• Long-form storytelling
• Instruction following / chat
• Any safety-critical application

Model Family

Model	Context	PPL	HF Repo
TinyTurk v2.7d	128	9.48	stunmuffin/TinyTurk-v2.7d
TinyTurk v2.7d-bigstories	128	3.99	stunmuffin/TinyTurk-v2.7d-bigstories
TinyTurk v2.7d-bigstories-256	256	3.60	stunmuffin/TinyTurk-v2.7d-bigstories-256
TinyTurk v2.7d-bigstories-384	384	3.69	this model

Citation

@misc{tinyturk_v27d_bigstories_384_2026,
  title = {TinyTurk v2.7d BigStories-384: Turkish 1.12M-parameter LM with 384-token context},
  author = {TinyTurk Project},
  year = {2026}
}

TinyTurk v2.7d BigStories-384 is an experimental research artifact. Not a product.