af-run demo — simulated agent behaviour scored by the real graders, statistics and gate; not measurements of the live apps.Improve · 07 · case.v1
Versioned suites with stable splits: train feeds the optimizer, val selects candidates, test is touched only at the promotion gate.
DataPilot · benchmark
datapilot/benchmark
150
BIRD Mini-Dev subset · execution accuracy
1 version · latest 073f02ffe9 · Sep 1
DataPilot · redteam
datapilot/redteam
116
seed attacks + accepted mutations
2 versions · latest 64cae93c6b · Sep 17
DataPilot · regression
datapilot/regression
52
mined from live failures + every fixed bug
3 versions · latest a73cb3cc34 · Oct 9
ReturnPilot · redteam
returnpilot/redteam
109
seed attacks + accepted mutations
2 versions · latest bdfef26e67 · Sep 15
ReturnPilot · regression
returnpilot/regression
19
mined from live failures + every fixed bug
2 versions · latest 2cb785f348 · Oct 2
ReturnPilot · scenario
returnpilot/scenario
30
hand-written multi-turn scenarios
1 version · latest e1bd2c24ce · Sep 1
Toy agent · redteam
toy/redteam
7
seed attacks + accepted mutations
1 version · latest 74ac2937dc · Sep 1
Toy agent · scenario
toy/scenario
12
hand-written multi-turn scenarios
1 version · latest 46b5f54e5f · Sep 1
| Case | Suite | Split | Origin | Tags / category | Hash |
|---|---|---|---|---|---|
| dp-bird-1031At present, calculate for the player's age who have a sprint speed of no less than 97 between 2013 to 2015. | datapilot/benchmark | test | seed | challengingeuropean_football_2bird | 1dcec9338d |
| dp-bird-1048What is the overall rating of the football player Gabriel Tamas in year 2011? | datapilot/benchmark | test | seed | simpleeuropean_football_2bird | 6efe250c58 |
| dp-bird-1094How much higher in percentage is Ariel Borysiuk's overall rating than that of Paulin Puel? | datapilot/benchmark | test | seed | challengingeuropean_football_2bird | f6ae061c74 |
| dp-bird-1107When was the first time did Kevin Constant have his highest crossing score? Give the date. | datapilot/benchmark | test | seed | moderateeuropean_football_2bird | 6d67224fdd |
| dp-bird-1136How many players had the highest potential score for crossing that preferred to use their left foots while attacking? | datapilot/benchmark | test | seed | moderateeuropean_football_2bird | 65a0d0bf75 |
| dp-bird-1153What is the disease patient '30609' diagnosed with. List all the date of laboratory tests done for this patient. | datapilot/benchmark | test | seed | simplethrombosis_predictionbird | 122b4d2db6 |
| dp-bird-1189What number of patients with a degree of thrombosis level 2 and ANA pattern of only S, have a level of anti-Cardiolip in antibody (IgM) 20% higher than average? | datapilot/benchmark | test | seed | challengingthrombosis_predictionbird | c5dbf419f0 |
| dp-bird-1225List and group all patients by sex for total bilirubin (T-BIL) level not within the normal range. | datapilot/benchmark | test | seed | moderatethrombosis_predictionbird | 66e6eb255c |
| dp-bird-125For loans contracts which are still running where client are in debt, list the district of the and the state the percentage unemployment rate increment from year 1995 to 1996. | datapilot/benchmark | test | seed | challengingfinancialbird | 548608484a |
| dp-bird-1256How many patients with a abnormal C-reactive protein don't have their data recorded? | datapilot/benchmark | test | seed | moderatethrombosis_predictionbird | 8f3404a328 |
| dp-bird-1270Among the patients who has a normal anti-scl70, how many of them are female and does not have any symptom? | datapilot/benchmark | test | seed | challengingthrombosis_predictionbird | d55123c2db |
| dp-bird-1322Among the events attended by more than 10 members of the Student_Club, how many of them are meetings? | datapilot/benchmark | test | seed | moderatestudent_clubbird | 96ceab4611 |
| dp-bird-1323List all the names of events that had an attendance of over 20 students but were not fundraisers. | datapilot/benchmark | test | seed | moderatestudent_clubbird | 6a9e8a6fca |
| dp-bird-1357State the date Connor Hilton paid his/her dues. | datapilot/benchmark | test | seed | simplestudent_clubbird | 0b6cb0dad8 |
| dp-bird-1394How many members of the Student_Club have major in 'Physics Teaching'? | datapilot/benchmark | test | seed | simplestudent_clubbird | e713e27840 |
| dp-bird-1480What was the gas consumption peak month for SME customers in 2013? | datapilot/benchmark | test | seed | moderatedebit_card_specializingbird | c758063938 |
| dp-bird-1500Please list the product description of the products consumed in September, 2013. | datapilot/benchmark | test | seed | simpledebit_card_specializingbird | d9cde2b907 |
| dp-bird-1506Please list the product descriptions of the transactions taken place in the gas stations in the Czech Republic. | datapilot/benchmark | test | seed | moderatedebit_card_specializingbird | 9ca07ab500 |
| dp-bird-152What is the average number of crimes committed in 1995 in regions where the number exceeds 4000 and the region has accounts that are opened starting from the year 1997? | datapilot/benchmark | test | seed | moderatefinancialbird | ee31cdda16 |
| dp-bird-159List all the withdrawals in cash transactions that the client with the id 3356 makes. | datapilot/benchmark | test | seed | simplefinancialbird | 136b0239f0 |
| dp-bird-168What percentage of clients who opened their accounts in the district with an average salary of over 10000 are women? | datapilot/benchmark | test | seed | moderatefinancialbird | 646e16b034 |
| dp-bird-197Calculate the average number of oxygen atoms in single-bonded molecules. | datapilot/benchmark | test | seed | moderatetoxicologybird | ec71a76d19 |
| dp-bird-215How many atoms with iodine and with sulfur type elements are there in single bond molecules? | datapilot/benchmark | test | seed | challengingtoxicologybird | f0799f575b |
| dp-bird-244Is the molecule with the most double bonds carcinogenic? | datapilot/benchmark | test | seed | moderatetoxicologybird | 8f603d9ac9 |
| dp-bird-282What is the ratio of Hydrogen elements in molecule ID TR006? List the ratio with its label. | datapilot/benchmark | test | seed | challengingtoxicologybird | 5f84cdb1cc |
| dp-bird-344List all the mythic rarity print cards banned in gladiator format. | datapilot/benchmark | test | seed | moderatecard_gamesbird | 6e6ffda732 |
| dp-bird-37What is the complete address of the school with the lowest excellence rate? Indicate the Street, City, Zip and State. | datapilot/benchmark | test | seed | moderatecalifornia_schoolsbird | 0d755485e1 |
| dp-bird-377How many cards with original type of "Summon - Angel" have subtype other than "Angel"? | datapilot/benchmark | test | seed | simplecard_gamesbird | 5172a7c03f |
| dp-bird-39What is the average number of test takers from Fresno schools that opened between 1/1/1980 and 12/31/1980? | datapilot/benchmark | test | seed | simplecalifornia_schoolsbird | 16c5be2598 |
| dp-bird-405How many Brazilian Portuguese translated sets are inside the Commander block? | datapilot/benchmark | test | seed | moderatecard_gamesbird | 2f1a37f295 |
| dp-bird-477Which of these artists have designed a card in the set Coldsnap, Jeremy Jarvis, Aaron Miller or Chippy? | datapilot/benchmark | test | seed | challengingcard_gamesbird | c527ab0ef8 |
| dp-bird-530List all the frame styles and cards Allen Williams worked on and find any banned cards if there are any. | datapilot/benchmark | test | seed | moderatecard_gamesbird | c65de9dbe4 |
| dp-bird-571For the user No.24, how many times is the number of his/her posts compared to his/her votes? | datapilot/benchmark | test | seed | moderatecodebase_communitybird | f91f5b4895 |
| dp-bird-578Which user made a post titled 'Understanding what Dassault iSight is doing?' and how much is the reputation of the user? | datapilot/benchmark | test | seed | moderatecodebase_communitybird | c654213b6a |
| dp-bird-629Calculate the ratio of votes in 2010 and 2011. | datapilot/benchmark | test | seed | simplecodebase_communitybird | efeca674d4 |
| dp-bird-634Among posts by Harvey Motulsky and Noah Snyder, which one has higher popularity? | datapilot/benchmark | test | seed | challengingcodebase_communitybird | 2ca76f7778 |
| dp-bird-671What is the display name of the user who acquired the first Autobiographer badge? | datapilot/benchmark | test | seed | simplecodebase_communitybird | 20c9023936 |
| dp-bird-745Give the publisher ID of Star Trek. | datapilot/benchmark | test | seed | simplesuperherobird | 3a0828dc56 |
| dp-bird-761Among the male superheroes, list the super hero names of superheroes with weight greater than the 79% average weight of all superheroes. | datapilot/benchmark | test | seed | moderatesuperherobird | 5147ce1097 |
| dp-bird-769Which superhero has the most durability published by Dark Horse Comics? | datapilot/benchmark | test | seed | challengingsuperherobird | 84366e83f4 |
| dp-bird-786How many heroes have the highest attribute value in strength? | datapilot/benchmark | test | seed | moderatesuperherobird | c751dfbeb7 |
| dp-bird-798What is the publisher for Hawkman, Karate Kid and Speedy? | datapilot/benchmark | test | seed | moderatesuperherobird | 538760e5f3 |
| dp-bird-847What is the surname of the driver with the best lap time in race number 19 in the second qualifying period? | datapilot/benchmark | test | seed | simpleformula_1bird | 7a1c835bed |
| dp-bird-85What is the Percent (%) Eligible Free (K-12) in the school administered by an administrator whose first name is Alusine. List the district code of the school. | datapilot/benchmark | test | seed | moderatecalifornia_schoolsbird | 3d4c452dd6 |
| dp-bird-884List the names of all races that occurred in the earliest recorded year and month. | datapilot/benchmark | test | seed | moderateformula_1bird | 7241e9812a |
| dp-bird-895What is the average lap time for Lewis Hamilton in the 2009 Malaysian Grand Prix? | datapilot/benchmark | test | seed | moderateformula_1bird | 4a77d4d5fa |
| dp-bird-944How much faster in percentage is the champion than the driver who finished the race last in the 2008 Australian Grand Prix? | datapilot/benchmark | test | seed | challengingformula_1bird | d76b77988a |
| dp-bird-945How many circuits are there in Adelaide, Australia? | datapilot/benchmark | test | seed | simpleformula_1bird | aa4c0747d7 |
| dp-bird-978How many times the circuits were held in Austria? Please give their location and coordinates. | datapilot/benchmark | test | seed | simpleformula_1bird | 1045e39b8a |
| dp-bird-989Who is the champion of the Canadian Grand Prix in 2008? Indicate his finish time. | datapilot/benchmark | test | seed | moderateformula_1bird | 409270a196 |
split = sha256(case_id) mod 10 → 0–5 train · 6–7 val · 8–9 test · red-team cases split by seed family · comparisons use only the intersection of cases.