Loading patterns…
Eval-Driven Development (Agent CI)(EDD)
The development-lifecycle discipline of building agents and prompts against evals first. Curated golden datasets (roughly 50 to 500 cases weighted toward known failure modes) are version-controlled alongside the prompts, run as a regression suite on every pull request, and merge or promotion is gated on metric thresholds. The pipeline pins model and judge versions so a silent provider-side model update is detected as a regression instead of being quietly absorbed, and stages its checks as lint, then offline eval, then a cost gate before a change can merge. Distinct from agent-observability-tracing, which is runtime telemetry rather than pre-merge gating, and from fixed public benchmarks, which are not team-owned regression suites in CI.
In 30 seconds
- What
- Version-controlled golden datasets run as regression suites on every PR, gating merge on metric thresholds with pinned model and judge versions.
- When to use
- Teams shipping agents where silent model updates must be detected, known failure modes need continuous coverage, and prompt changes require measurable validation.
- Watch out
- Golden datasets become stale or misaligned with real production failures if not actively maintained and weighted toward current failure modes.
Ask the AI expert about this pattern
Opens the assistant with your question prefilled. You review it before sending.
Eval-Driven Development (Agent CI): Overview
The development-lifecycle discipline of building agents and prompts against evals first. Curated golden datasets (roughly 50 to 500 cases weighted toward known failure modes) are version-controlled alongside the prompts, run as a regression suite on every pull request, and merge or promotion is gated on metric thresholds. The pipeline pins model and judge versions so a silent provider-side model update is detected as a regression instead of being quietly absorbed, and stages its checks as lint, then offline eval, then a cost gate before a change can merge. Distinct from agent-observability-tracing, which is runtime telemetry rather than pre-merge gating, and from fixed public benchmarks, which are not team-owned regression suites in CI.
- Version-controlled golden datasets (~50-500 cases) weighted toward known failure modes
- Regression suite runs on every pull request; merge gated on metric thresholds
- Pinned model and judge versions so silent provider updates surface as regressions
- Staged CI pipeline: lint, then offline eval, then cost gate before merge
- LLM-as-judge and code-based scorers combined per failure dimension
- Prompt and dataset changes reviewed together as a single diff
Get the Agent Evals field guide
All 25 agent evaluation methods condensed into one guide: which benchmark measures what, when a public score misleads you, and how to build evals out of your own failures. The link arrives with your confirmation, alongside the weekly Agent Architect.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.
References
The papers, specifications, and repositories this pattern is based on.
From the engineer behind this catalog
Find out what your evals miss
Measuring an agent is harder than shipping one, and most suites stay green while production drifts. Have your evaluation setup reviewed end to end: what you measure today, what you cannot see yet, and the regressions your current suite would let through.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September