File size: 2,131 Bytes
3ad96d4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
license: apache-2.0
library_name: transformers
tags:
- laya
- system-one
- calibrated-decisions
- rlcd
- structured-decisions
- typed-decisions
- benchmark
metrics:
- accuracy
- brier_score
model-index:
- name: laya-typed-decisions
results:
- task:
type: text-classification
name: System One Decision Benchmark
dataset:
type: LocalLLaMA/typed-decisions
name: Typed Decisions
metrics:
- type: accuracy
value: 0.758
- type: brier_score
value: 0.059
---
# Laya (Fine-Tuned on Typed-Decisions Benchmark)
This is **Laya** fine-tuned on the 1,200 training cases (6,000 decisions) of the independent [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) benchmark.
On the official 400-case test set (2,000 decisions across Agent Trace Observability, Customer Service, Invoice Processing, and Security Incidents), it achieves **0.758 Accuracy**, outperforming **TypeSafe Jev 1.13.0 (0.727)** and surpassing the benchmark's **Teacher Self-Agreement ceiling (0.735)**.
## Head-to-Head Benchmark Results
| Model | Kind | Accuracy | Soft Acc | Brier Score | ECE | Score MAE | Within 1 Level | Latency (p50) | Cost/Case |
|---|---|---|---|---|---|---|---|---|---|
| **Laya (Ours)** | **fine-tuned** | **0.758** | **0.563** | **0.059** | **0.135** | **0.258** | **0.986** | **15.1 ms** | **$0.00 (Self-Hosted)** |
| TypeSafe Jev 1.13.0 | general | 0.727 | 0.580 | 0.148 | 0.144 | 0.391 | 0.952 | 710 ms | $0.0004 (API) |
| ModernBERT-base (149M) | specialist | 0.646 | 0.542 | 0.119 | 0.179 | 0.444 | 0.931 | 349 ms | $0.00 |
| Teacher Self-Agreement | ceiling | 0.735 | - | - | - | - | - | - | - |
## Installation & Quickstart
```bash
pip install laya
```
```python
import laya
# Load the fine-tuned model directly from Hugging Face
agent = laya.load("convaiinnovations/laya-typed-decisions")
# Evaluate any workflow state and typed questions in a single forward pass
result = agent.predict(state, questions)
print(result["answers"])
```
## License
Apache 2.0. Developed by [Convai Innovations](https://huggingface.co/convaiinnovations).
|