af-run demo — simulated agent behaviour scored by the real graders, statistics and gate; not measurements of the live apps.Improve · 07 · case.v1
Versioned suites with stable splits: train feeds the optimizer, val selects candidates, test is touched only at the promotion gate.
DataPilot · benchmark
datapilot/benchmark
150
BIRD Mini-Dev subset · execution accuracy
1 version · latest 073f02ffe9 · Sep 1
DataPilot · redteam
datapilot/redteam
116
seed attacks + accepted mutations
2 versions · latest 64cae93c6b · Sep 17
DataPilot · regression
datapilot/regression
52
mined from live failures + every fixed bug
3 versions · latest a73cb3cc34 · Oct 9
ReturnPilot · redteam
returnpilot/redteam
109
seed attacks + accepted mutations
2 versions · latest bdfef26e67 · Sep 15
ReturnPilot · regression
returnpilot/regression
19
mined from live failures + every fixed bug
2 versions · latest 2cb785f348 · Oct 2
ReturnPilot · scenario
returnpilot/scenario
30
hand-written multi-turn scenarios
1 version · latest e1bd2c24ce · Sep 1
Toy agent · redteam
toy/redteam
7
seed attacks + accepted mutations
1 version · latest 74ac2937dc · Sep 1
Toy agent · scenario
toy/scenario
12
hand-written multi-turn scenarios
1 version · latest 46b5f54e5f · Sep 1
| Case | Suite | Split | Origin | Tags / category | Hash |
|---|---|---|---|---|---|
| dp-demo-chinook-length-priceIs there a correlation between track length and price? | datapilot/regression | test | seed | bird-styleanalysissandbox | 1686268899 |
| dp-demo-chinook-monthly-2013What were the monthly sales in 2013? | datapilot/regression | test | seed | bird-styletime-seriesevidence | 9494cf40da |
| dp-reg-bird-1144Please state the finishing rate and curve score of the player who has the heaviest weight. | datapilot/regression | test | seed | simpleeuropean_football_2bird | cc5eef04d6 |
| dp-reg-bird-1239Name the ID and age of patient with two or more laboratory examinations which show their hematoclit level exceeded the normal range. | datapilot/regression | test | seed | challengingthrombosis_predictionbird | 0919991e73 |
| dp-reg-bird-1526For the customer who paid 634.8 in 2012/8/25, what was the consumption decrease rate from Year 2012 to 2013? | datapilot/regression | test | seed | challengingdebit_card_specializingbird | efb7baceaf |
| dp-reg-bird-462What's the Italian name of the set of cards with "Ancestor's Chosen" is in? | datapilot/regression | test | seed | moderatecard_gamesbird | de86ca87cb |
| dp-reg-bird-687How many comments were added to the post with the highest score? | datapilot/regression | test | seed | simplecodebase_communitybird | ec185822ed |
| dp-reg-bird-751List down at least five superpowers of male superheroes. | datapilot/regression | test | seed | moderatesuperherobird | f2bfcf9eb3 |
| dp-reg-bird-79Between San Diego and Santa Barbara, which county offers the most number of schools that does not offer physical building? Indicate the amount. | datapilot/regression | test | seed | moderatecalifornia_schoolsbird | c5061fb1fa |
| dp-reg-bird-98Among the accounts who have approved loan date in 1997, list out the accounts that have the lowest approved amount and choose weekly issuance statement. | datapilot/regression | test | seed | moderatefinancialbird | e5d7705125 |
| dp-reg-mined-5e086f75Mined: which · genre · longest (status failure) | datapilot/regression | test | mined | minedwhich-genre-longest-status-failure | bc8803d00a |
split = sha256(case_id) mod 10 → 0–5 train · 6–7 val · 8–9 test · red-team cases split by seed family · comparisons use only the intersection of cases.