Loading patterns…
Terminal-Bench(TB)
Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.
In 30 seconds
- What
- Poses hard command line tasks in isolated containers, each carrying a human-written solution and tests that decide afterwards whether the work was actually finished.
- When to use
- You are evaluating agents that do infrastructure, build or debugging work in a shell, and you need correctness decided rather than judged.
- Watch out
- The score names a model and a harness together: the same weights under a weaker scaffold give up exactly where a stronger one reads the error and recovers.
Ask the AI expert about this pattern
Opens the assistant with your question prefilled. You review it before sending.
Terminal-Bench: Overview
Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.
- One container per task, so runs cannot contaminate each other
- Every task carries a human-written reference solution
- Verified by tests rather than by inspecting the transcript
- Drawn from real infrastructure, data and debugging work
- Deliberately hard: frontier systems remain well short of solving it
- Harness-agnostic, so the agent and the scaffold are measured together
Get the Agent Evals field guide
All 25 agent evaluation methods condensed into one guide: which benchmark measures what, when a public score misleads you, and how to build evals out of your own failures. The link arrives with your confirmation, alongside the weekly Agent Architect.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.
Also known as: CLI agent benchmark, Command line agent evaluation
References
The papers, specifications, and repositories this pattern is based on.
From the engineer behind this catalog
Find out what your evals miss
Measuring an agent is harder than shipping one, and most suites stay green while production drifts. Have your evaluation setup reviewed end to end: what you measure today, what you cannot see yet, and the regressions your current suite would let through.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September