Muster werden geladen…
⌨️
Terminal-Bench(TB)
Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.
Komplexität: highEvaluierung und Monitoring
In 30 Sekunden
- Was
- Poses hard command line tasks in isolated containers, each carrying a human-written solution and tests that decide afterwards whether the work was actually finished.
- Wann einsetzen
- You are evaluating agents that do infrastructure, build or debugging work in a shell, and you need correctness decided rather than judged.
- Achtung
- The score names a model and a harness together: the same weights under a weaker scaffold give up exactly where a stronger one reads the error and recovers.
Fragen Sie den KI-Experten zu diesem Pattern
Öffnet den Assistenten mit vorbereiteter Frage. Sie prüfen sie vor dem Senden.
Terminal-Bench: Überblick
Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.
- One container per task, so runs cannot contaminate each other
- Every task carries a human-written reference solution
- Verified by tests rather than by inspecting the transcript
- Drawn from real infrastructure, data and debugging work
- Deliberately hard: frontier systems remain well short of solving it
- Harness-agnostic, so the agent and the scaffold are measured together
The Agent Architect
Ein Pattern, ein Tradeoff, eine Produktionspanne. Ein kurzes wöchentliches Briefing für alle, die agentische Systeme bauen.
Wöchentliche E-Mail, Abmeldung mit einem Klick. Ihre Adresse wird nur für das Briefing verwendet.
Auch bekannt als: CLI agent benchmark, Command line agent evaluation
Quellen
Die Veröffentlichungen, Spezifikationen und Repositories, auf denen dieses Muster beruht.