thechristyjo/laya-model-routing-500

Laya Model Routing — 500-row scaling run (best checkpoint)

Laya (421M) fine-tuned to classify a user query into one of six model-routing classes, so an application can pick the right class of free OpenRouter models (single forward pass):

class	meaning
fast_cheap	greetings, basic lookups, formatting
general_mid	everyday reasoning, most chat/Q&A
top_reasoning	hard multi-step reasoning, math, planning
code_specialist	writing, debugging, reviewing code
security_specialist	vulnerabilities, secure coding, privacy, auth
long_context	long documents / many turns

Results (held-out set: 100 disjoint queries, judge-labeled)

Scaling curve on the same eval set:

train rows	model	choice_accuracy	ECE ↓
0	base Laya (zero-shot)	0.630	0.169
200	laya-model-routing	0.820	0.125
500	this checkpoint	0.910	0.074
1000	laya-model-routing-1000	0.870	0.087

Majority-class baseline: 0.270 · random: 0.167.

Per-class recall: security_specialist 1.000 · general_mid 0.960 · fast_cheap 0.926 · code_specialist 0.909 · long_context 0.667 · top_reasoning 0.000 (3 training rows — never predicted).

Fixes the v1 weakness: general_mid recall 0.600 → 0.960.

Training

• Data: 500 queries from ai-mitra/llm-router-dataset (shuffled seed 42, rows 300–799), labeled by glm-5.3 as judge via Z.AI
• Recipe: RLCD (policy gradient + proper scoring rule + cross-entropy), DDP on 2× T4, 8 epochs, temperature calibration on a holdout slice
• Full pipeline and experiment log: github.com/lunakicks/laya-model-routing

Usage

import laya

agent = laya.load("thechristyjo/laya-model-routing-500", device="cpu")
questions = {
    "model_class": {
        "type": "choice",
        "instructions": "Which model class should handle this task?",
        "criteria": {
            "fast_cheap": "simple, short, low-stakes queries",
            "general_mid": "everyday reasoning, most chat/Q&A",
            "top_reasoning": "hard multi-step reasoning, math, planning",
            "code_specialist": "writing, debugging, reviewing code",
            "security_specialist": "security, privacy, auth tasks",
            "long_context": "long documents or many turns",
        },
    }
}
print(agent.predict("Write a function to detect a linked-list cycle.", questions)
      ["answers"]["model_class"]["choice"])

Limitations

• This checkpoint shipped an out-of-range choice temperature (7.07, clamped to 5 by laya at load with a warning) — measured ECE is the best of the series, but re-calibrate before tightening any confidence gate above 0.5
• top_reasoning has 3 training rows — never predicted; needs targeted augmentation
• long_context recall 0.667 — trained on short one-liners about summarization, not actual long inputs
• English-only; judge labels carry the judge's "cheapest competent class" policy