af-run demo — simulated agent behaviour scored by the real graders, statistics and gate; not measurements of the live apps.Issue 03 · Monitor · 01
2 agents under test. Last nightly 10 days ago — every rate shown with its 95% interval.
01 · Multi-agent text-to-SQL analyst
DataPilotLast nightly pass rate
73%
95% 60%–83% · n=52
27 nights
Attack success
5.0%
2/40 · 1%–17%
Cost / case
$0.0016
list price · paid $0
p50 latency
4.9 s
p95 10.3 s
Hard failures
0
must_not + canary
02 · Returns agent with human approval
ReturnPilotLast nightly pass rate
76%
95% 62%–85% · n=49
regression suite · active profile · Wilson 95% band
ledger · 2026-10-09 · caps from SPEC §15.1
28 nights
Attack success
4.3%
2/47 · 1%–14%
Cost / case
$0.0012
list price · paid $0
p50 latency
3.5 s
p95 7.6 s
Hard failures
0
must_not + canary
gated · human-approved · audit-logged
Toy agent v1 → v2efficiency
test 0% → 100% · Sohail-5678 · Oct 1
ReturnPilot v4 → v3rollback
Sohail-5678 · Sep 26
ReturnPilot v3 → v4efficiency
test 54% → 54% · Sohail-5678 · Sep 25
DataPilot v2 → v3quality
test 55% → 72% · Sohail-5678 · Sep 23
DataPilot v2 → v2rejected
test 57% → 60% · Sohail-5678 · Sep 21
ReturnPilot v2 → v3efficiency
test 46% → 54% · Sohail-5678 · Sep 17
Cohen's κ vs blind human labels · gate needs κ ≥ 0.6, n ≥ 50
κ0.86
agreement 93% · n=56
κ0.58
agreement 79% · n=38
drafted from failing live traces · human-accepted only
12
drafts waiting for a human