Evaluate / Attack /
Improve / Gate
Traces, graders, attacks and prompts shaping the agents that ship after dark — every improvement proven with statistics before a human lets it go live.
Most portfolios build an agent. This is the harness around two of them — the part companies actually struggle with.
495
Cases under test
8 suites
94
Attack seeds
10 OWASP categories
0.86
Judge agreement (Cohen’s κ) *
n=56 blind labels
6
Gated promotions *
88 runs graded
* from the demo history: simulated DataPilot/ReturnPilot behaviour scored by AgentForge's real graders, statistics and gate — not measurements of the live apps.
Chapter 00 — the loop
DataPilot and ReturnPilot send redacted traces to the control plane. Failures become draft test cases a human reviews. GitHub Actions runs the suites and attacks in throwaway containers, graders score every case, and the optimizer proposes better prompts — which go live only through the gate.
beyond “it looked fine”
Deterministic end-state and trajectory graders, hard must-not safety properties, canary tokens, and an LLM judge that only counts once Cohen’s kappa against blind human labels reaches 0.6.
guardrails proven, not assumed
Ten OWASP-mapped attack categories, 60+ hand-written seeds, deterministic and LLM mutations, and per-layer attribution that shows which defence actually stopped each attack.
reflective prompt evolution
A GEPA-style optimizer reads failed traces, rewrites one component at a time, and keeps a Pareto front — while locked safety fields, keep-blocks and a deny-list stop it from weakening the agent.
statistics, safety, then a human
A candidate ships only if its test-split gain has a bootstrap interval above zero and McNemar p < 0.05 (or it is 20% cheaper at equal quality), the attack rate does not rise, and an admin approves the diff.
How it was built
The always-on part is light: Next.js route handlers on Vercel and Neon Postgres store traces, profiles and results. The heavy part is bursty: GitHub Actions runs the Python runner image with a throwaway Postgres per job — reproducible by image tag, target commit, profile version and suite hash. Total cost: $0.
Targets · Render
DataPilot API
trace.v1 → · ← profile.v1
ReturnPilot API
trace.v1 → · ← profile.v1
Control plane · Vercel
Next.js dashboard + /api/v1
ingest · profiles · runs · reviews · gate
Neon Postgres
traces · cases · runs · results · profiles
Execution plane · Actions
agentforge-runner image
graders · red team · optimizer
Throwaway Postgres
target eval adapter @ commit
Honest limitations
With 50 test cases you can only prove large gains — so the gate says “not proven”, never “no effect”.