MaziyarPanahi/Qwen3.5-4B-Jev-Judge

Qwen3.5-4B-Judge

A 4B open judge for two everyday LLM checks: is this RAG answer grounded in its context? and is this request or response unsafe? It returns a calibrated probability for each option, not free text.

Qwen3.5-4B + an Unsloth decision head (LoRA r=16), trained on 20,000 rows of the two public Jev decision datasets, using the human / source labels in the datasets.

Results

Every model is scored on the same held-out rows, gold = the human / source label. Jev 1.14 was called through OpenRouter's Decisions endpoint (TypeSafe only, no fallbacks) with the same state, instructions and criteria.

Test set	Rows	This model	Jev 1.14
RAG claim checks (Jev-RAG test split)	1,873	87.9%	82.1%
Safety decisions (Jev-Safety test split)	2,162	90.7%	82.3%
Both, the tasks it was trained for	4,035	89.4%	82.2%
MedHallu PubMed answers, never trained on	2,000	78.4%	67.9%

On MedHallu (1,000 PubMed questions, each with the real answer and a hallucinated one), how often does the judge give the real answer a higher "grounded" probability than the fake one? This model: 855/1,000. Jev 1.14: 916/1,000 (43 ties). Jev's default cutoff is strict on PubMed (it rejects many real answers), so its accuracy there understates how well it ranks. Tune the threshold for your data.

Training

• Data: 10,000 random rows from each dataset's train split (the trivial quality question excluded), seed 3407.
• Recipe: Unsloth FastDecisionModel, bf16 LoRA r=16, lr 2e-4, batch 8 x 4, 1 epoch, max length 4,096, calibrated on 400 held-out training rows.
• Hardware: 1x NVIDIA H200 on Hugging Face Jobs, 22.5 min of training ($1.87 at the listed $5.00/h).

Files

• The repo root is the merged model: Qwen3.5-4B with the LoRA folded in (16-bit), plus the decision head (joint_head.safetensors) and its configs. It loads directly, no base model or PEFT needed.
• adapter/ is the original LoRA adapter + decision head, for when you already have Qwen3.5-4B.
• joint_schema_model.py is Cloudflare's Apache-2.0 Clef reference code, shipped by Unsloth's export. In our checks it did not reproduce Unsloth's answers on every row, so use Unsloth (below), which we verified against the training run.

The calibration temperature is folded into the head.

Use

With Unsloth, the merged model at the root (or the adapter, with the commented line):

from unsloth import FastDecisionModel

model, tokenizer = FastDecisionModel.from_pretrained("MaziyarPanahi/Qwen3.5-4B-Jev-Judge", max_seq_length=4096, load_in_4bit=False)
# model, tokenizer = FastDecisionModel.from_pretrained("MaziyarPanahi/Qwen3.5-4B-Jev-Judge", subfolder="adapter", max_seq_length=4096, load_in_4bit=False)
FastDecisionModel.for_inference(model)
question = {"decision": {"type": "choice",
    "instructions": "Is every factual claim in `assistant_response` supported by `authoritative_context`?",
    "criteria": {"true": "Every factual claim in the response is supported by the context.",
                 "false": "The response contains at least one unsupported or contradicted claim."}}}
state = '{"authoritative_context": "...", "assistant_response": "..."}'
print(FastDecisionModel.predict(model, tokenizer, state, question))

Limits

• It was trained in-domain: the RAG and safety test rows come from the same sources as its training rows. Jev 1.14 is a general decision model answering zero-shot.
• PubMed answers are a transfer test, not medical validation. Do not use it to make clinical decisions.
• Data: RAGTruth via wandb/RAGTruth-processed (MIT, upstream review noted on the dataset card) and NVIDIA Aegis 2.0 (CC-BY-4.0). MedHallu (UTAustin-AIHealth/MedHallu) was used for evaluation only.