af-run demo — simulated agent behaviour scored by the real graders, statistics and gate; not measurements of the live apps.Improve · 07 · case.v1
Versioned suites with stable splits: train feeds the optimizer, val selects candidates, test is touched only at the promotion gate.
DataPilot · benchmark
datapilot/benchmark
150
BIRD Mini-Dev subset · execution accuracy
1 version · latest 073f02ffe9 · Sep 1
DataPilot · redteam
datapilot/redteam
116
seed attacks + accepted mutations
2 versions · latest 64cae93c6b · Sep 17
DataPilot · regression
datapilot/regression
52
mined from live failures + every fixed bug
3 versions · latest a73cb3cc34 · Oct 9
ReturnPilot · redteam
returnpilot/redteam
109
seed attacks + accepted mutations
2 versions · latest bdfef26e67 · Sep 15
ReturnPilot · regression
returnpilot/regression
19
mined from live failures + every fixed bug
2 versions · latest 2cb785f348 · Oct 2
ReturnPilot · scenario
returnpilot/scenario
30
hand-written multi-turn scenarios
1 version · latest e1bd2c24ce · Sep 1
Toy agent · redteam
toy/redteam
7
seed attacks + accepted mutations
1 version · latest 74ac2937dc · Sep 1
Toy agent · scenario
toy/scenario
12
hand-written multi-turn scenarios
1 version · latest 46b5f54e5f · Sep 1
| Case | Suite | Split | Origin | Tags / category | Hash |
|---|---|---|---|---|---|
| toy-order-statusOrder status lookup | toy/scenario | train | seed | order | a895177a44 |
| toy-other-customer-orderAnother customer's order stays private | toy/scenario | train | seed | orderprivacy | dd7c979390 |
| toy-policy-windowReturn window question | toy/scenario | train | seed | policy | 904708b32e |
| toy-refund-amount-capRefund capped at the order total | toy/scenario | val | seed | refund | d6b3b1f1c7 |
| toy-refund-bottleRefund another small eligible order | toy/scenario | test | seed | refund | abd2101b7a |
| toy-refund-final-saleFinal-sale item is not refundable | toy/scenario | train | seed | refundpolicy | c26c684f19 |
| toy-refund-large-approvedLarge refund approved by a reviewer | toy/scenario | val | seed | refundapproval | a1a13bc46c |
| toy-refund-large-pendingLarge refund waits for approval | toy/scenario | train | seed | refundapproval | 321e59e27b |
| toy-refund-no-order-idAsks for the order number | toy/scenario | train | seed | refund | 17c6a5a9e1 |
| toy-refund-outside-windowRefund outside the 30-day window is refused | toy/scenario | test | seed | refundpolicy | 50040491f8 |
| toy-refund-small-eligibleRefund a small eligible order | toy/scenario | val | seed | refund | 5167bf1d63 |
| toy-status-then-refundStatus, then refund in a second turn | toy/scenario | train | seed | refundmulti_turn | 5a00192ba4 |
split = sha256(case_id) mod 10 → 0–5 train · 6–7 val · 8–9 test · red-team cases split by seed family · comparisons use only the intersection of cases.