Independent test: Julia-1 as an agent step guard (vs Laya) — fast, but never escalates to a human
Julia-1's card notes that unfamiliar domains and multi-step reasoning can cause misses. I tested one such domain: judging an AI coding agent between steps (continue / finish / ask a human / stop). This is a common use for decision models, and I'd previously run Laya on the same set.
Setup:
- Data: sinhal/agent-step-guard-bench, 61 hand-written agent transcripts × 4 questions (1 choice, 3 yes/no).
- Input: each transcript as plain text. The three Laya checkpoints were re-run on the identical input.
- Run: zero-shot, CPU,
load_model(..., strict_encoding=True, max_length=8192, head_length=512), named-question API.
| system | next_action | mean (4 questions) | CPU ms per transcript |
|---|---|---|---|
| always the most common answer | 44.3% | 67.6% | |
| Julia-1 | 23.0% | 58.2% | 246 |
| Laya typed-decisions | 32.8% | 54.1% | 1,151 |
| Laya multilingual | 24.6% | 45.5% | 432 |
| Laya English | 18.0% | 49.6% | 1,078 |
What I saw:
- Fast, and the best mean of the four. It's about 4.7× faster than Laya typed-decisions on CPU.
- It never picked
ask_human: 0 of 61, where 20 are correct. On the blocked cases (missing API key, ambiguous spec, required review) it chosefinish, often at 0.9+. One example: "I need the live secret key to proceed" gavefinishat 0.99.ask_humanstayed at ≤4% probability throughout. - It never flags risky actions, e.g.
git push --force origin mainorDELETE FROM users;on prod. The risky flag fired 0% of the time, and its ranking is at chance (AUROC 0.49). - High confidence that doesn't track correctness: mean top probability 0.81 when wrong vs 0.78 when right.
So for this domain, the card's caveat holds: it's good at picking among clear labels, but it doesn't recognise "I need a human here" situations. Fine-tuning on escalation examples would likely be needed for this use. I'd be curious whether Dumont behaves differently.
Possible issue on my side: the ask_human option ID contains an underscore. Its description ("the agent needs human input or approval before going further") was supplied, so I don't think the ID matters, but happy to re-run with a different ID if you think it does.
Code, raw outputs and the full table: https://github.com/Zephyr4772/laya-zeroshot-agent-guard-eval (§6)