Sieve-27B-v2

Sieve-27B-v2 is a decision model. You give it a piece of text or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question. It reads the state once, scores the options directly, and never generates text.

It is a full fine-tune of Qwen/Qwen3.8-27B (revision 1d4bf0f2) with a pointer head, and it stores one calibration temperature per question category. The code is at github.com/sthanika-ai/Sieve.

Summary

Backbone	Qwen/Qwen3.8-27B (revision 1d4bf0f2)
Design	pointer head
Trained weights	full fine-tune of the 64 decoder layers (24.35B parameters), and a pointer head
Question types	choice, noul and score (trained on choice and noul)
Decision Index	public score 68.02 on our own run of the 0.3 suite; not on the leaderboard yet
Temperature	a global T = 0.999 and 25 per question category (0.928 to 1.127)
Token limits	16,384 tokens for the state and 16,384 for each question with its options
GPU memory	about 52 GB in bf16 (50.3 to 53.1 GB on the CUDA-graph server, 52.2 GB on the eager server)

How it works

• Backbone. Only the text part of Qwen3.8-27B is used; the model cannot produce text.
• Full fine-tune. All 64 decoder layers were trained; the token embeddings and the final norm are the backbone's. The release stores the trained layers in full as a PEFT adapter (modules_to_save), with a zero rank-1 LoRA on the embeddings. sieve 0.3.2 and later copy these layers into the backbone in place, so the model needs about the backbone's memory to load.
• Readout. A pointer head scores the end of each option against the decision point. A calibrated softmax gives the probabilities.
• Isolation. Each question runs as its own row from the state's cache, so questions never see each other.

type	options	returns
choice	the caller's keys, optionally described (up to 255)	a probability per key
noul	yes / no	P(yes)
score	ordered levels	a probability per level

Use

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0   # optional on A100 / H100: the fused convolution kernel

from sieve import load_sieve, decide

m = load_sieve("sthanika-ai/Sieve-27B-v2")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
       {"department": {"type": "choice", "instructions": "Which team handles this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

• Loading. Load it with sieve 0.3.2 or later, which loads a full fine-tune in place; the adapter alone is not a text-generation model. The backbone, Qwen/Qwen3.8-27B, is downloaded on first use; our runs used revision 1d4bf0f2.
• GPU memory. In bf16, 50.3 to 53.1 GB was in use on the CUDA-graph server and 52.2 GB on the eager server during our latency runs. One 80 GB GPU is enough; longer states need more.
• Token limits. States and questions up to 16,384 tokens each, as in training; sieve reads these limits from training_config.json. Longer requests are refused, never truncated. The Decision Index engine declares larger limits (a 65,536-token state and a 32,768-token question), and every request of the Decision Index run below fit within them.
• Server. sieve-serve --model sthanika-ai/Sieve-27B-v2 --graphs.
• Decision Index. python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-27B-v2 --out runs/Sieve-27B-v2.
• causal-conv1d. Our training, Decision Index and latency runs were on NVIDIA B300 GPUs, for which causal-conv1d has no kernels, so they used transformers' PyTorch convolution. On A100 or H100 the fused kernel is faster; its probabilities can differ slightly from ours.
• Temperature per category. head.pt stores a global T = 0.999 and 25 temperatures (0.928 to 1.127), fitted by NLL on the calibration split (132,395 questions, outside the checkpoint-selection sample). A question's category is its type, its number of options (2, 3-4, 5-9, 10-25, 26+) and the kind of state (empty, text or JSON). A question takes the temperature of its most specific category in the table (choice:3-4:text, then choice:3-4, then choice), else the global T. Temperatures never change an answer, and temperature=1.0 gives raw probabilities.

Results

Decision Index

Sieve-27B-v2 scores 68.02 (raw index 75.61, breadth 67.38) on the public 0.3 suite of the Decision Index, on our own run with the official kit at f5a6e4f (0.3) and its scorer.

• Coverage. All 140,178 scoreable requests over 43 benchmarks were answered, with no truncation, no unsupported request and no error. The index averages 37 of the benchmarks.
• How the run was made. One complete run of the 0.3 suite on two NVIDIA B300 GPUs, in six shards.
• Results. sthanika-ai/Sieve-27B-v2-decision-index-results.
• Not on the leaderboard yet. The score is self-reported. The board ranks by the Full score: the public benchmarks count 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%.

Decision Index 0.3, public suite	Knowledge & Reasoning	Language Understanding	Retrieval & Classification	Tools & Automation	Arts & Human Taste
68.02	56.8	66.0	73.4	85.8	58.9

The index and area scores are chance-corrected: 0 is random guessing and 100 is perfect. Below, the score is in each benchmark's own metric and the skill is chance-corrected.

All 43 benchmarks

benchmark	metric	score	skill (×100)	requests
ACOS	per-review F1	0.483	46.7	1,565
Amazon ESCI	macro-F1	0.613	51.5	5,000
ANLI	macro-F1	0.752	62.8	3,200
API-Bank	accuracy	0.890	88.8	508
BANKING77	macro-F1	0.899	89.8	3,080
BBH fixed-option tasks	accuracy	0.882	82.9	5,507
BFCL	case exact accuracy	0.960	94.7	1,694
BPoMP	accuracy	0.980	95.8	5,000
BRIGHT	nDCG@10	0.461	39.0	220
cfcolor	accuracy	0.648	31.8	5,000
ChessBench	accuracy	0.247	18.0	5,000
CLadder	accuracy	0.933	86.5	5,000
CLINC150+OOS	macro-F1	0.909	90.8	5,500
ContractNLI	macro-F1	0.834	76.0	123
CRUXEval	accuracy	0.818	71.0	570
FinEntity	macro-F1	0.932	90.0	979
GPQA Diamond	accuracy	0.495	32.6	196
GSM8K	accuracy	0.720	66.5	2,638
Habermas Machine	accuracy	0.657	50.2	1,676
HellaSwag	accuracy	0.970	96.0	10,042
HLE	accuracy	0.104	0.0	501
Home appliance simulator	case exact accuracy	1.000	100.0	88
HoVer claim verification	accuracy	0.880	76.0	4,000
Humicroedit	accuracy	0.601	20.2	2,628
iSarcasmEval	Sarcasm F1 · track A, English	0.473	32.2	4,600
MMLU-Pro	accuracy	0.624	57.7	12,032
MuSR	accuracy	0.883	81.4	752
New Yorker caption matching	accuracy	0.780	72.5	528
NLI4CT	macro-F1	0.854	71.6	5,500
PhishNChips phishing decisions	accuracy	0.967	93.3	2,000
POP909-CL	accuracy	0.833	83.0	2,000
RAGTruth response-level hallucination	F1 on hallucinated class	0.740	46.1	2,700
SATA-Bench	case exact accuracy	0.429	42.1	1,650
ToolRet	nDCG@10	0.685	63.7	685
VAST	macro-F1	0.781	67.1	3,006
When2Call MCQ	accuracy	0.848	79.7	3,652
WinoGrande	accuracy	0.952	90.4	1,267
ARC-Challenge (not in the index)	accuracy	0.962	94.9	1,172
ARC-Easy (not in the index)	accuracy	0.985	98.0	2,376
MMLU (not in the index)	accuracy	0.823	76.4	14,033
RouterBench (not in the index)	selected quality (quality objective)	0.798	52.6	10,000
SGD/SGD-X (not in the index)	macro-F1	0.465	10.9	2,500
SimpleBench (not in the index)	accuracy	0.000	0.0	10

Calibration

The calibration split chose the checkpoint and fitted the temperatures.

split	questions	accuracy	NLL	Brier	ECE
calibration, per-category temperatures (shipped)	132,395	0.9263	0.1874	0.0991	0.0020
calibration, one global T = 0.999	132,395	0.9263	0.1876	0.0991	0.0024
calibration, raw (T = 1)	132,395	0.9263	0.1876	0.0991	0.0025

In a 2-fold check inside the calibration split (fit on one half, score the other), the per-category table scored NLL 0.1875 and ECE 0.0026, against 0.1876 and 0.0029 for a single T.

Latency

• Per Decision Index request. In-process wall time, one request at a time, eager, over the 140,620 requests of the 0.3 run (NVIDIA B300): median 136 ms, mean 347 ms, 80th percentile 322 ms, 95th percentile 918 ms. Six engines (three per GPU) ran at once, so these are upper bounds. This is not the maintainers' RTX PRO 6000 measurement.
• Cached state. Server-reported p50 on one NVIDIA B300 (bf16, CUDA graphs), for one choice question on a 1,433-token state:

options	2	10	50
state cached (ms)	95.6	97.2	97.6
state new (ms)	208.5	208.6	206.8

• By option count. For one choice question on a 41-token state, which is below the server's 64-token caching threshold, so every request ran the state again. Server p50 in ms, CUDA graphs / eager:

options	2	5	10	25	50	100	255
graphs	119.5	120.2	122.2	134.0	153.4	190.2	345.4
eager	174.2	170.8	168.2	170.8	170.6	100.8	254.4

Training

• Recipe.
  • Full fine-tune of the 64 decoder layers (24,353,196,544 parameters); the token embeddings and the final norm stay frozen. The pointer head (256-dimensional projections, fp32) is trained from scratch, alone for the first 100 steps.
  • AdamW, lr 4e-6 for the backbone and 5e-5 for the head, weight decay 0.01 on weight matrices, gradient clipping at 1.0, 5% linear warm-up then cosine decay to 0.1×, 1 epoch, 128 records per step.
  • bf16 compute with fp32 master weights, gradient reduction and optimizer state; FSDP2 with one unit per decoder layer; gradient checkpointing.
  • States and questions up to 16,384 tokens each; longer records are dropped, never truncated (none were).
  • 3,133 steps, 27.55 hours (55.1 GPU-hours) on two NVIDIA B300 GPUs.
  • Permuted choice options, "none of the above" augmentation (as the answer, as a wrong option and in minimal pairs) and irrelevant distractor options, with a cross-entropy loss.
• Selection. The checkpoint with the highest accuracy on a fixed 6,004-question calibration sample, with NLL breaking ties. This rule was fixed before training, and it chose the last step, 3,133. The benchmarks were never used to choose.
• Temperatures. Fitted after selection on the calibration split outside the selection sample: the global T, then one T per category with at least 200 calibration questions.
• Data. 400,985 training records, with 65,959 for calibration and 201,027 for test (60/10/30).
  • Sources. Public datasets and generated decision records.
  • Not published. The training files are not part of this release; the SHA-256 of each file is in training_config.json.

Limitations

• score questions. The training data has only choice and noul questions. score questions are answered by the same pointer head, but the model was not trained on them, and they take the global temperature.

Files

file	contents
adapter_model.safetensors, adapter_config.json	the 64 fine-tuned decoder layers in full (PEFT modules_to_save, bf16) and a zero rank-1 LoRA on the embeddings
head.pt	the pointer head and the calibrated temperatures (global and per category)
tokenizer.json, tokenizer_config.json	Qwen3.8-27B's tokenizer, unchanged
training_config.json, training_metrics.json, train.log	the recipe, the run and its log
result.json	the calibration, Decision Index and latency numbers above, with the calibration metrics per category
provenance.json	SHA-256 of every file and of the training data, and library versions

License

Apache-2.0, see LICENSE.