af-run demo — simulated agent behaviour scored by the real graders, statistics and gate; not measurements of the live apps.Improve · 06 · GEPA-lite
Reflective prompt evolution on failed traces, one component at a time, with a Pareto front — and a gate that only statistics, safety and a human can open.
ReturnPilot · from v3
opt-returnpilot-2026-10-05
candidates
7
on front
1
editor ✗
1
started Oct 5 · updated 14 days ago
Toy agent · from v1
opt-toy-2026-09-29
candidates
8
on front
1
editor ✗
2
started Sep 29 · finished 19 days ago
DataPilot · from v2
opt-datapilot-2026-09-18
candidates
11
on front
1
editor ✗
2
started Sep 18 · finished last month
ReturnPilot · from v2
opt-returnpilot-2026-09-13
candidates
10
on front
2
editor ✗
2
started Sep 13 · finished last month
every gate decision with its evidence · rollbacks included
Test split · n=2
0% → 100%
gain +100 pts, 95% CI +100 to +100
Paired test
p=0.50
McNemar exact · b=0 c=2
Verdict
efficiency
n=2: 80% power only for ≥0 pts · P(detect +10)=0%
Hard safety
0 must_not, 0 canary on test + red-team
Red team
successes 4→0, high-severity 4→0 (n=7)
Quality path
gain +1.000 (95% CI +1.000 to +1.000), McNemar p=0.5
Efficiency pathpath used
cost -22%, p95 -35%, test 0%→100%
Reliability
pass^2 0%→100%
Judge
judge not calibrated: rubric results excluded from the gate
Red-team ASR
57.1% → 0.0%
High-severity wins
4 → 0
Cost / case
$0.0001 (−22%)
pass^2
0% → 100%
Nightly paired test: pass rate dropped after v4 (shorter history drops policy context).
gate runs: candidate
Test split · n=13
54% → 54%
gain ±0 pts, 95% CI +0 to +0
Paired test
p=1.00
McNemar exact · b=0 c=0
Verdict
efficiency
n=13: 80% power only for ≥0 pts · P(detect +10)=0%
Hard safety
0 must_not, 0 canary on test + red-team
Red team
successes 2→2, high-severity 0→0 (n=47)
Quality path
gain +0.000 (95% CI +0.000 to +0.000), McNemar p=1
Efficiency pathpath used
cost -23%, p95 +14%, test 54%→54%
Reliability
pass^2 54%→54%
Judge
judge calibrated; rubric pass rate 96%→96%
Red-team ASR
4.3% → 4.3%
High-severity wins
0 → 0
Cost / case
$0.0010 (−23%)
pass^2
54% → 54%
Test split · n=60
55% → 72%
gain +17 pts, 95% CI +7 to +28
Paired test
p=0.006
McNemar exact · b=1 c=11
Verdict
quality
n=60: 80% power only for ≥17 pts · P(detect +10)=43%
Hard safety
0 must_not, 0 canary on test + red-team
Red team
successes 3→2, high-severity 1→1 (n=40)
Quality pathpath used
gain +0.167 (95% CI +0.067 to +0.283), McNemar p=0.00635
Efficiency path
cost +0%, p95 -1%, test 55%→72%
Reliability
pass^2 55%→70%
Judge
judge not calibrated: rubric results excluded from the gate
Red-team ASR
7.5% → 5.0%
High-severity wins
1 → 1
Cost / case
$0.0024 (+0%)
pass^2
55% → 70%
Test split · n=60
57% → 60%
gain +3 pts, 95% CI +0 to +8
Paired test
p=0.50
McNemar exact · b=0 c=2
Verdict
not proven
n=60: 80% power only for ≥16 pts · P(detect +10)=46%
Hard safety
0 must_not, 0 canary on test + red-team
Red team
successes 3→3, high-severity 1→1 (n=40)
Quality path
gain +0.033 (95% CI +0.000 to +0.083), McNemar p=0.5
Efficiency path
cost +22%, p95 +26%, test 57%→60%
Reliability
pass^2 57%→57%
Judge
judge not calibrated: rubric results excluded from the gate
Red-team ASR
7.5% → 7.5%
High-severity wins
1 → 1
Cost / case
$0.0029 (+22%)
pass^2
57% → 57%
Test split · n=13
46% → 54%
gain +8 pts, 95% CI −15 to +31
Paired test
p=1.00
McNemar exact · b=1 c=2
Verdict
efficiency
n=13: 80% power only for ≥0 pts · P(detect +10)=1%
Hard safety
0 must_not, 0 canary on test + red-team
Red team
successes 6→2, high-severity 2→0 (n=47)
Quality path
gain +0.077 (95% CI -0.154 to +0.308), McNemar p=1
Efficiency pathpath used
cost -58%, p95 -49%, test 46%→54%
Reliability
pass^2 46%→54%
Judge
judge calibrated; rubric pass rate 89%→96%
Red-team ASR
12.8% → 4.3%
High-severity wins
2 → 0
Cost / case
$0.0013 (−58%)
pass^2
46% → 54%