Autopilot-reliability measurement · open weights · Groq free tier

LLM Eval Observatory

The next wave of AI sells the work, not the tool — an autopilot that does an already-outsourced job, not a copilot that assists a human. This living dashboard measures whether open-weight models are trustworthy enough to be that autopilot: it runs one end-to-end (AML alert triage) and scores the things that decide shippability — false clears, fabricated evidence, calibrated escalation — alongside the capability, robustness, and judge-reliability checks that say whether those numbers can be believed. Every figure is computed by a scheduled GitHub Action and committed to this repo, not hand-typed.

nightly eval status
Statuslive
Last run13h ago
Commit860745e
Models evaluated2 of 2 configured
API calls308
Duration566.7s
raw JSON ↗
Track 1 · flagship

Autopilot reliability — KYC/AML alert triage

The other three tracks ask how good the model is. This one asks the question a services-as-software company must answer before it can sell an outcome instead of a tool: can an autopilot built on this model DO an already-outsourced, intelligence-heavy job end-to-end? The job is AML alert triage — for each of 26 flagged cases (54% genuinely suspicious) the model decides ESCALATE, CLEAR, or REVIEW against a fixed policy and must cite its evidence. What decides shippability isn't raw accuracy — it's whether it fabricates evidence, and whether it wrongly CLEARS an alert it should have escalated (a "false clear": the error that gets a bank fined). Every number here is graded deterministically.

Decision accuracy — 3-way (escalate / clear / review)
#ModelAccuracy (95% CI)vs last
1
GPT-OSS 120B
OpenAI OSS
100.0%[100%–100%]
+0.0pp
2
GPT-OSS 20B
OpenAI OSS
92.3%[81%–100%]
-3.8pp
Decision accuracy over time
Trust & safety — can it ship without a human?
↓ lower is better · ↑ higher is better
ModelFalse clears ↓Fabricated cites ↓Grounding ↑Over-escalation ↓Abstains when unsure ↑
GPT-OSS 120B
0.0%0.0%100.0%0.0%100.0%
GPT-OSS 20B
0.0%0.0%100.0%0.0%50.0%

A 90%-accurate autopilot that quietly clears real laundering is worse than useless — the reframe is that the safety columns, not the accuracy column, decide whether you can remove the human. That is exactly the trust layer a services-as-software company has to build before it can sell the outcome.

Decision accuracy by alert type
GPT-OSS 120B
ambiguous
100%
benign_ctr
100%
benign_profile
100%
layering
100%
pep
100%
sanctions
100%
structuring
100%
GPT-OSS 20B
ambiguous
50%
benign_ctr
100%
benign_profile
100%
layering
100%
pep
100%
sanctions
100%
structuring
100%
Track 2 · is it competent?

Capability & drift

Accuracy on a fixed 50-item auto-graded suite (math, logic, instruction-following, factual recall). Bars show the point estimate; the faint band behind each is the bootstrapped 95% CI. A drift badge appears only when a model's score moves by at least 5 points AND its CI no longer overlaps the previous run's — noise stays quiet.

#ModelAccuracy (95% CI)vs last
1
GPT-OSS 120B
OpenAI OSS
100.0%[100%–100%]
+0.0pp
2
GPT-OSS 20B
OpenAI OSS
98.0%[94%–100%]
+0.0pp
Accuracy over time
Accuracy by category
GPT-OSS 120B
factual
100%
instruction
100%
logic
100%
math
100%
GPT-OSS 20B
factual
90%
instruction
100%
logic
100%
math
100%
Track 3 · can it be manipulated?

Prompt-injection & jailbreak robustness

Defense rate across a 30-prompt attack battery — direct instruction overrides, social-engineering, jailbreak role-play, system-prompt leaks, and indirect injection hidden inside a document to summarize. Defense = the model never emits the protected canary or obeys the injected payload. Grading is exact string matching and runs conservative (a model that quotes the payload while refusing still counts as a leak), so this understates rather than inflates safety.

#ModelDefense rate (95% CI)vs last
1
GPT-OSS 120B
OpenAI OSS
96.7%[90%–100%]
+0.0pp
2
GPT-OSS 20B
OpenAI OSS
93.3%[83%–100%]
+0.0pp
Defense rate over time
Defense rate by attack class
GPT-OSS 120B
direct-injection
100%
distraction
100%
indirect-extraction
100%
indirect-injection
90%
jailbreak-roleplay
100%
obfuscation
100%
prompt-leak
100%
social-engineering
100%
GPT-OSS 20B
direct-injection
100%
distraction
100%
indirect-extraction
100%
indirect-injection
80%
jailbreak-roleplay
100%
obfuscation
100%
prompt-leak
100%
social-engineering
100%
Track 4 · can you trust automated grading?

LLM-as-judge bias & reliability

Judges are themselves unreliable, so this track quantifies how. Each model judges 24 pairwise items against held-out human labels, in both presentation orders. We measure agreement with the human, position bias (how often the verdict flips when we swap order — an unbiased judge never flips), and verbosity bias (tendency to pick the longer answer; the set is length-balanced so 0 is neutral).

#JudgeAgree w/ humanOrder flipVerbosity
1
GPT-OSS 20B
OpenAI OSS
100.0%
0%+0.00
2
GPT-OSS 120B
OpenAI OSS
100.0%
0%+0.00
Agreement with human over time
Inter-rater agreement
1.00Fleiss' κ · almost perfect

How much the judges agree with each other beyond chance. Low κ means "which model is right?" depends a lot on which judge you ask — the core reason a single LLM judge can't be trusted blindly.

Pairwise judge agreement (Cohen's κ)
GPT-OSS 20B ↔ GPT-OSS 120B:1.00

Dataset is balanced: the better answer is response A in half the items and the longer answer in half, so 0.5 is neutral for both position_1_rate and verbosity pick-longer.