af-run demo — simulated agent behaviour scored by the real graders, statistics and gate; not measurements of the live apps.Improve · 07 · case.v1
Versioned suites with stable splits: train feeds the optimizer, val selects candidates, test is touched only at the promotion gate.
DataPilot · benchmark
datapilot/benchmark
150
BIRD Mini-Dev subset · execution accuracy
1 version · latest 073f02ffe9 · Sep 1
DataPilot · redteam
datapilot/redteam
116
seed attacks + accepted mutations
2 versions · latest 64cae93c6b · Sep 17
DataPilot · regression
datapilot/regression
52
mined from live failures + every fixed bug
3 versions · latest a73cb3cc34 · Oct 9
ReturnPilot · redteam
returnpilot/redteam
109
seed attacks + accepted mutations
2 versions · latest bdfef26e67 · Sep 15
ReturnPilot · regression
returnpilot/regression
19
mined from live failures + every fixed bug
2 versions · latest 2cb785f348 · Oct 2
ReturnPilot · scenario
returnpilot/scenario
30
hand-written multi-turn scenarios
1 version · latest e1bd2c24ce · Sep 1
Toy agent · redteam
toy/redteam
7
seed attacks + accepted mutations
1 version · latest 74ac2937dc · Sep 1
Toy agent · scenario
toy/scenario
12
hand-written multi-turn scenarios
1 version · latest 46b5f54e5f · Sep 1
| Case | Suite | Split | Origin | Tags / category | Hash |
|---|---|---|---|---|---|
| dp-bird-1028In Scotland Premier League, which away team won the most during the 2010 season? | datapilot/benchmark | train | seed | challengingeuropean_football_2bird | a328cd8e06 |
| dp-bird-1088Please list the names of the players whose volley score and dribbling score are over 70. | datapilot/benchmark | train | seed | moderateeuropean_football_2bird | 13148a382e |
| dp-bird-1110Tell the build Up play passing class for "FC Lorient" on 2010/2/22. | datapilot/benchmark | train | seed | moderateeuropean_football_2bird | e4443a6cd3 |
| dp-bird-1147Please name one player whose overall strength is the greatest. | datapilot/benchmark | train | seed | simpleeuropean_football_2bird | b965b3058e |
| dp-bird-1162How many female patients who came at the hospital in 1997 was immediately followed at the outpatient clinic? | datapilot/benchmark | train | seed | moderatethrombosis_predictionbird | 93facf6f26 |
| dp-bird-1164How many of the patients with the most serious thrombosis cases examined in 1997 are women? | datapilot/benchmark | train | seed | moderatethrombosis_predictionbird | 1ffa2f2339 |
| dp-bird-1169What is the ratio of male to female patients among all those with abnormal uric acid counts? | datapilot/benchmark | train | seed | challengingthrombosis_predictionbird | 2bf951d355 |
| dp-bird-118For loan amount less than USD100,000, what is the percentage of accounts that is still running with no issue. | datapilot/benchmark | train | seed | moderatefinancialbird | 803d1f614f |
| dp-bird-12Among the schools with an SAT excellence rate of over 0.3, what is the highest eligible free rate for students aged 5-17? | datapilot/benchmark | train | seed | moderatecalifornia_schoolsbird | a439017f23 |
| dp-bird-1220Provide all ID, sex and birthday of patients whose urea nitrogen (UN) just within the borderline of passing? | datapilot/benchmark | train | seed | simplethrombosis_predictionbird | c7f2be0212 |
| dp-bird-1252Among the patients with a normal Ig G level, how many of them have symptoms? | datapilot/benchmark | train | seed | moderatethrombosis_predictionbird | c0772ca1d3 |
| dp-bird-136Between 1/1/1995 and 12/31/1997, how many loans in the amount of at least 250,000 per account that chose monthly statement issuance were approved? | datapilot/benchmark | train | seed | moderatefinancialbird | 40f8d41970 |
| dp-bird-1381List the name of students that have attended more than 7 events. | datapilot/benchmark | train | seed | moderatestudent_clubbird | 0adad1b9e8 |
| dp-bird-1409Mention the total expense used on 8/20/2019. | datapilot/benchmark | train | seed | simplestudent_clubbird | 79487b215c |
| dp-bird-1411State what kind of expenses that Sacha Harrison incurred? | datapilot/benchmark | train | seed | simplestudent_clubbird | 130adfb227 |
| dp-bird-1427What are the budget category of the events located at MU 215 and a guest speaker type with a 0 budget spent? | datapilot/benchmark | train | seed | moderatestudent_clubbird | 5d874a148d |
| dp-bird-1435List the names of closed event as "game" that was closed from 3/15/2019 to 3/20/2020. | datapilot/benchmark | train | seed | moderatestudent_clubbird | f58cf15247 |
| dp-bird-1471What is the ratio of customers who pay in EUR against customers who pay in CZK? | datapilot/benchmark | train | seed | simpledebit_card_specializingbird | 9ab425f551 |
| dp-bird-149Please list the account types that are not eligible for loans, and the average income of residents in the district where the account is located exceeds $8000 but is no more than $9000. | datapilot/benchmark | train | seed | challengingfinancialbird | 91cc22f4f1 |
| dp-bird-1493In February 2012, what percentage of customers consumed more than 528.3? | datapilot/benchmark | train | seed | simpledebit_card_specializingbird | 54e50528b4 |
| dp-bird-1521For all the transactions happened during 8:00-9:00 in 2012/8/26, how many happened in CZE? | datapilot/benchmark | train | seed | moderatedebit_card_specializingbird | 54ce41b1a4 |
| dp-bird-228How much of the hydrogen in molecule TR206 is accounted for? Please provide your answer as a percentage with four decimal places. | datapilot/benchmark | train | seed | moderatetoxicologybird | 940d52e25e |
| dp-bird-231Which bond type accounted for the majority of the bonds found in molecule TR010 and state whether or not this molecule is carcinogenic? | datapilot/benchmark | train | seed | challengingtoxicologybird | 9f0a4065e1 |
| dp-bird-236What are the bond type and the atoms of the bond ID of TR001_6_9? | datapilot/benchmark | train | seed | moderatetoxicologybird | d840d18025 |
| dp-bird-31What is the eligible free rate of the 10th and 11th schools with the highest enrolment for students in grades 1 through 12? | datapilot/benchmark | train | seed | moderatecalifornia_schoolsbird | c7cdd1eb3f |
| dp-bird-345For artifact type of cards that do not have multiple faces on the same card, state its legalities status for vintage play format. | datapilot/benchmark | train | seed | moderatecard_gamesbird | dd2ed9451a |
| dp-bird-356How many cards have infinite power? | datapilot/benchmark | train | seed | simplecard_gamesbird | bae6b48fac |
| dp-bird-36Under whose administration is the school with the highest number of students scoring 1500 or more on the SAT? Indicate their full names. | datapilot/benchmark | train | seed | challengingcalifornia_schoolsbird | 2b9ac9de8c |
| dp-bird-366What is the rule of playing card "Benalish Knight"? | datapilot/benchmark | train | seed | simplecard_gamesbird | 41e3f40402 |
| dp-bird-391Among the Artifact cards, which are black color and comes with foreign languague translation? | datapilot/benchmark | train | seed | moderatecard_gamesbird | 4c097337b0 |
| dp-bird-397What is the mana cost of cards with a normal layout, a 2003 frame version, with a black border color, and available in paper and mtgo? | datapilot/benchmark | train | seed | moderatecard_gamesbird | 0d0198a959 |
| dp-bird-528List the names of all the cards in the set Hour of Devastation and find the formats in which these cards are legal. | datapilot/benchmark | train | seed | challengingcard_gamesbird | eda30309e8 |
| dp-bird-557Among the posts with a score of over 5, what is the percentage of them being owned by an elder user? | datapilot/benchmark | train | seed | moderatecodebase_communitybird | f7c168cd98 |
| dp-bird-592How many users are awarded with more than 5 badges? | datapilot/benchmark | train | seed | simplecodebase_communitybird | 09a3fbd3fe |
| dp-bird-633Which post by slashnick has the most answers count? State the post ID. | datapilot/benchmark | train | seed | moderatecodebase_communitybird | 348bef2cfd |
| dp-bird-669When did 'chl' cast its first vote in a post? | datapilot/benchmark | train | seed | simplecodebase_communitybird | c80c9fb25b |
| dp-bird-682Which is the most valuable post in 2010? Please give its id and the owner's display name. | datapilot/benchmark | train | seed | moderatecodebase_communitybird | 3511970d92 |
| dp-bird-717Please list all the superpowers of 3-D Man. | datapilot/benchmark | train | seed | simplesuperherobird | 0b5f27f402 |
| dp-bird-728Rank superheroes from Marvel Comics by their eye color popularity, starting with the most common color. | datapilot/benchmark | train | seed | moderatesuperherobird | 70180b2916 |
| dp-bird-743What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code? Indicate how many of the said superheroes were published by Marvel Comics. | datapilot/benchmark | train | seed | challengingsuperherobird | 5b541d775a |
| dp-bird-764What are the superpowers of heroes with ID 1? | datapilot/benchmark | train | seed | simplesuperherobird | 6dc1178d33 |
| dp-bird-773Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero. | datapilot/benchmark | train | seed | challengingsuperherobird | 2d2de5b55e |
| dp-bird-794Which hero was the fastest? | datapilot/benchmark | train | seed | moderatesuperherobird | 6a59777348 |
| dp-bird-866Who was the player that got the lap time of 0:01:27 in the race No. 161? Show his introduction website. | datapilot/benchmark | train | seed | moderateformula_1bird | b74ad2f471 |
| dp-bird-881For the drivers who took part in the race in 1983/7/16, what's their race completion rate? | datapilot/benchmark | train | seed | moderateformula_1bird | 01ea4b5604 |
| dp-bird-937What's the finish time for the driver who ranked second in 2008's AustChineseralian Grand Prix? | datapilot/benchmark | train | seed | simpleformula_1bird | 5f5fc5a0ed |
| dp-bird-962From 2000 to 2005, what percentage of drivers who were born before 1985 and the lap numbers were over 50? | datapilot/benchmark | train | seed | challengingformula_1bird | d50f8eddc8 |
| dp-bird-977From race no. 50 to 100, how many finishers have been disqualified? | datapilot/benchmark | train | seed | simpleformula_1bird | e253d9dd30 |
| dp-bird-981On what year did the youngest driver had his first qualifying race? Also state the name, date and time of the race. | datapilot/benchmark | train | seed | moderateformula_1bird | a34019f6b9 |
| dp-bird-988List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985. | datapilot/benchmark | train | seed | challengingformula_1bird | 8e9fc7bdc1 |
split = sha256(case_id) mod 10 → 0–5 train · 6–7 val · 8–9 test · red-team cases split by seed family · comparisons use only the intersection of cases.