Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
SeaWolf-AI 
posted an update 2 days ago
Post
3123
AX-Ray: Safety Diagnostics for AI/AX Models

AI models can no longer be evaluated only by capability scores. As models move into public services, enterprise workflows, scientific research, and administrative decision support, we need a second layer of evaluation: whether the model behaves safely, structurally, and consistently under real deployment conditions.

VIDRAFT AX-Ray is a public AI/AX safety diagnostic initiative powered by FINAL-Bench Diagnostics. AX-Ray evaluates models across a structured guideline framework, including model-level safety, AX deployment readiness, and agent/service operation risks. The public diagnostic catalog contains 117 diagnostic items, mapped to legal, regulatory, ethical, and religious-law governance contexts so that safety review can be discussed in a form closer to real institutional responsibility.

A central finding of AX-Ray is causal leakage: a structural defect where information that should not influence an earlier reasoning state appears to affect model behavior. AX-Ray presents a public case of diagnosing, reproducing, and demonstrating causal leakage in two general-purpose public models. This matters because such defects are not exposed by ordinary benchmark scores. A model can appear capable while still carrying hidden safety or integrity risks.

Explore the live leaderboard, diagnostic reports, and public dataset here:

- AX-Ray Space: FINAL-Bench/AX-RAY
- AX-Ray Dataset: FINAL-Bench/AX-RAY
- Technical Article: https://huggingface.co/blog/FINAL-Bench/ax-ray

AX-Ray is intended as a practical guideline for moving AI evaluation beyond “how smart is the model?” toward “can this model be trusted, governed, and deployed safely?”

The causal-leakage framing is valuable because ordinary capability benchmarks can miss structure-level failures. A practical next step would be to publish the minimal reproducibility harness and a per-diagnostic pass/fail trace, so deployment teams can compare models and audit regressions without relying on a single aggregate score.

·

Thanks — that's the direction we're building toward.

We agree on the premise: a single aggregate score is the wrong unit for a deployment decision. That's why every diagnosis already produces a per-item result across the 117-item catalog rather than one number — the aggregate is a summary of that, not a substitute for it.

Exposing those per-diagnostic traces publicly is on the roadmap. We'd rather ship it once coverage is uniform across the leaderboard than publish partial traces that invite the wrong comparisons.

On the harness: the probe sets are held out deliberately — publishing them would contaminate the behaviours they measure. What we can open is the layer above them: item schemas, category distributions, and the pass/fail criteria per axis. That should be enough to audit regressions without making the benchmark self-defeating.

Appreciate the push. It's the right one.