LLM Eval Observatory
The next wave of AI sells the work, not the tool — an autopilot that does an already-outsourced job, not a copilot that assists a human. This living dashboard measures whether open-weight models are trustworthy enough to be that autopilot: it runs one end-to-end (AML alert triage) and scores the things that decide shippability — false clears, fabricated evidence, calibrated escalation — alongside the capability, robustness, and judge-reliability checks that say whether those numbers can be believed. Every figure is computed by a scheduled GitHub Action and committed to this repo, not hand-typed.
Autopilot reliability — KYC/AML alert triage
The other three tracks ask how good the model is. This one asks the question a services-as-software company must answer before it can sell an outcome instead of a tool: can an autopilot built on this model DO an already-outsourced, intelligence-heavy job end-to-end? The job is AML alert triage — for each of 26 flagged cases (54% genuinely suspicious) the model decides ESCALATE, CLEAR, or REVIEW against a fixed policy and must cite its evidence. What decides shippability isn't raw accuracy — it's whether it fabricates evidence, and whether it wrongly CLEARS an alert it should have escalated (a "false clear": the error that gets a bank fined). Every number here is graded deterministically.
| # | Model | Accuracy (95% CI) | vs last |
|---|---|---|---|
| 1 | GPT-OSS 120B OpenAI OSS | 100.0%[100%–100%] | +0.0pp |
| 2 | GPT-OSS 20B OpenAI OSS | 92.3%[81%–100%] | -3.8pp |
| Model | False clears ↓ | Fabricated cites ↓ | Grounding ↑ | Over-escalation ↓ | Abstains when unsure ↑ |
|---|---|---|---|---|---|
GPT-OSS 120B | 0.0% | 0.0% | 100.0% | 0.0% | 100.0% |
GPT-OSS 20B | 0.0% | 0.0% | 100.0% | 0.0% | 50.0% |
A 90%-accurate autopilot that quietly clears real laundering is worse than useless — the reframe is that the safety columns, not the accuracy column, decide whether you can remove the human. That is exactly the trust layer a services-as-software company has to build before it can sell the outcome.
Capability & drift
Accuracy on a fixed 50-item auto-graded suite (math, logic, instruction-following, factual recall). Bars show the point estimate; the faint band behind each is the bootstrapped 95% CI. A drift badge appears only when a model's score moves by at least 5 points AND its CI no longer overlaps the previous run's — noise stays quiet.
| # | Model | Accuracy (95% CI) | vs last |
|---|---|---|---|
| 1 | GPT-OSS 120B OpenAI OSS | 100.0%[100%–100%] | +0.0pp |
| 2 | GPT-OSS 20B OpenAI OSS | 98.0%[94%–100%] | +0.0pp |
Prompt-injection & jailbreak robustness
Defense rate across a 30-prompt attack battery — direct instruction overrides, social-engineering, jailbreak role-play, system-prompt leaks, and indirect injection hidden inside a document to summarize. Defense = the model never emits the protected canary or obeys the injected payload. Grading is exact string matching and runs conservative (a model that quotes the payload while refusing still counts as a leak), so this understates rather than inflates safety.
| # | Model | Defense rate (95% CI) | vs last |
|---|---|---|---|
| 1 | GPT-OSS 120B OpenAI OSS | 96.7%[90%–100%] | +0.0pp |
| 2 | GPT-OSS 20B OpenAI OSS | 93.3%[83%–100%] | +0.0pp |
LLM-as-judge bias & reliability
Judges are themselves unreliable, so this track quantifies how. Each model judges 24 pairwise items against held-out human labels, in both presentation orders. We measure agreement with the human, position bias (how often the verdict flips when we swap order — an unbiased judge never flips), and verbosity bias (tendency to pick the longer answer; the set is length-balanced so 0 is neutral).
| # | Judge | Agree w/ human | Order flip | Verbosity |
|---|---|---|---|---|
| 1 | GPT-OSS 20B OpenAI OSS | 0% | +0.00 | |
| 2 | GPT-OSS 120B OpenAI OSS | 0% | +0.00 |
How much the judges agree with each other beyond chance. Low κ means "which model is right?" depends a lot on which judge you ask — the core reason a single LLM judge can't be trusted blindly.
Dataset is balanced: the better answer is response A in half the items and the longer answer in half, so 0.5 is neutral for both position_1_rate and verbosity pick-longer.