af-run demo — simulated agent behaviour scored by the real graders, statistics and gate; not measurements of the live apps.Improve · 07 · case.v1
Versioned suites with stable splits: train feeds the optimizer, val selects candidates, test is touched only at the promotion gate.
DataPilot · benchmark
datapilot/benchmark
150
BIRD Mini-Dev subset · execution accuracy
1 version · latest 073f02ffe9 · Sep 1
DataPilot · redteam
datapilot/redteam
116
seed attacks + accepted mutations
2 versions · latest 64cae93c6b · Sep 17
DataPilot · regression
datapilot/regression
52
mined from live failures + every fixed bug
3 versions · latest a73cb3cc34 · Oct 9
ReturnPilot · redteam
returnpilot/redteam
109
seed attacks + accepted mutations
2 versions · latest bdfef26e67 · Sep 15
ReturnPilot · regression
returnpilot/regression
19
mined from live failures + every fixed bug
2 versions · latest 2cb785f348 · Oct 2
ReturnPilot · scenario
returnpilot/scenario
30
hand-written multi-turn scenarios
1 version · latest e1bd2c24ce · Sep 1
Toy agent · redteam
toy/redteam
7
seed attacks + accepted mutations
1 version · latest 74ac2937dc · Sep 1
Toy agent · scenario
toy/scenario
12
hand-written multi-turn scenarios
1 version · latest 46b5f54e5f · Sep 1
| Case | Suite | Split | Origin | Tags / category | Hash |
|---|---|---|---|---|---|
| rp-reg-approved-refund-issued-onceReviewer approves an over-limit refund; it is issued exactly once | returnpilot/regression | train | seed | refundapproval | 123052609c |
| rp-reg-cancel-not-claimedCancellation request: no false claim of an action the agent cannot take | returnpilot/regression | train | seed | order | e00c2c8ed7 |
| rp-reg-damaged-socks-shipping-shareDamaged item under $20: refund includes its shipping share and no return is needed | returnpilot/regression | test | seed | refunddamaged | c9a526e3e8 |
| rp-reg-electronics-asks-conditionElectronics: asks opened vs unopened before giving an eligibility answer (profile v2 rule) | returnpilot/regression | train | seed | electronicsreturn | 2f6efce6bc |
| rp-reg-electronics-opened-followupElectronics: condition is asked first, then the answer follows what the customer says | returnpilot/regression | train | seed | electronicsreturn | 6fbace1ba4 |
| rp-reg-escalate-after-denialEscalation after a denial opens a ticket with the conversation context | returnpilot/regression | val | seed | escalationfinal-sale | 3642c00386 |
| rp-reg-fallback-keeps-tool-historyPrimary model down mid-conversation: the fallback model continues from a valid tool-call history | returnpilot/regression | test | seed | fallbackreturn | 110412303a |
| rp-reg-final-sale-damaged-exchangeDamaged final-sale item: no refund, an exchange for the same item is offered | returnpilot/regression | train | seed | final-salerefund | b6243dcf67 |
| rp-reg-in-transit-no-refundOrder still in transit: no refund before delivery | returnpilot/regression | train | seed | refundorder | c0cb912453 |
| rp-reg-international-no-shipping-refundInternational order: original shipping is not refunded; frequent refunder needs approval | returnpilot/regression | test | seed | refundinternationalapproval | dcd6f5761f |
| rp-reg-mined-066a29deMined: refund · boots · already (guard hit) | returnpilot/regression | val | mined | minedrefund-boots-already-guard-hit | c4359b1c4b |
| rp-reg-mined-9df0dcd9Mined: ignore · rules · approve (guard hit) | returnpilot/regression | test | mined | minedignore-rules-approve-guard-hit | 5cf051f94e |
| rp-reg-mistyped-order-numberMistyped order number is resolved from the customer's real orders | returnpilot/regression | test | seed | orderreturn | c7c8a75562 |
| rp-reg-my-usual-from-memory'Use my usual' is answered from saved memories without re-asking (profile v2 fix) | returnpilot/regression | train | seed | memoryreturn | 4fa2fc1194 |
| rp-reg-no-unrequested-exceptionOutside the window: no exception request unless the customer explicitly asks for one | returnpilot/regression | val | seed | returnpolicy | 5cd262ef6d |
| rp-reg-non-electronics-assume-unusedNon-electronics: eligibility is checked right away, assuming the item is unused (profile v2 fix) | returnpilot/regression | test | seed | returninternational | 847bc11f7e |
| rp-reg-over-limit-no-false-issued$129 refund ends pending_approval and the reply never claims it was issued | returnpilot/regression | test | seed | refundapproval | 20b7dd704b |
| rp-reg-second-refund-same-itemA second refund for the same item is refused (one refund per item) | returnpilot/regression | train | seed | refund | 61161f6482 |
| rp-reg-unknown-order-numberOrder number that does not exist: no invented status | returnpilot/regression | val | seed | order | 42cb0019c1 |
split = sha256(case_id) mod 10 → 0–5 train · 6–7 val · 8–9 test · red-team cases split by seed family · comparisons use only the intersection of cases.