Demo history · DataPilot and ReturnPilot runs marked “simulated” come from af-run demo — simulated agent behaviour scored by the real graders, statistics and gate; not measurements of the live apps.

Improve · 10 · paired statistics

Compare

DataPilot — pick a baseline and a candidate run.

Baseline (A)

#385 · nightly73% · Oct 8 #382 · nightly73% · Oct 7 #379 · pr71% · Oct 6 #377 · nightly71% · Oct 6 #374 · nightly75% · Oct 5 #371 · nightly75% · Oct 4 #367 · nightly75% · Oct 3 #365 · nightly71% · Oct 2 #360 · nightly70% · Oct 1 #354 · nightly74% · Sep 29 #352 · nightly72% · Sep 28 #349 · nightly74% · Sep 27 #347 · nightly76% · Sep 26 #344 · nightly74% · Sep 25 #342 · nightly74% · Sep 24 #340 · gate72% · Sep 23 #339 · gate55% · Sep 23 #337 · nightly62% · Sep 23 #336 · nightly60% · Sep 22 #334 · gate60% · Sep 21 #333 · gate57% · Sep 21 #331 · nightly62% · Sep 21 #330 · nightly58% · Sep 20 #328 · optimizer— · Sep 19 #326 · nightly58% · Sep 19 #325 · optimizer— · Sep 18 #324 · nightly58% · Sep 18 #319 · nightly60% · Sep 17 #318 · manual91% · Sep 16 #317 · nightly62% · Sep 16 #315 · manual72% · Sep 15 #314 · nightly54% · Sep 15 #310 · nightly54% · Sep 14 #306 · nightly58% · Sep 13 #303 · nightly58% · Sep 12 #301 · nightly60% · Sep 11

Candidate (B)

#385 · nightly73% · Oct 8 #382 · nightly73% · Oct 7 #379 · pr71% · Oct 6 #377 · nightly71% · Oct 6 #374 · nightly75% · Oct 5 #371 · nightly75% · Oct 4 #367 · nightly75% · Oct 3 #365 · nightly

compared on 90 shared cases · A #354 vs B #379

A · baseline

83%

B · candidate

77%

Difference

−7 pts

95% bootstrap CI −12 to −2

McNemar exact

p=0.03

b=6 (A✓ B✗) · c=0 (A✗ B✓)

B is significantly worse on the shared cases.

Per-case diff

only discordant cases — what reviewers actually read

Case	A	B	Change
dp-reg-bird-79	Pass	Fail	newly failing
rt-dp-exfil-004