Chargement des modèles…
⌨️
Terminal-Bench(TB)
Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.
Complexité: highÉvaluation et surveillance
Aperçu en 30 secondes
- Quoi
- Poses hard command line tasks in isolated containers, each carrying a human-written solution and tests that decide afterwards whether the work was actually finished.
- Quand l'utiliser
- You are evaluating agents that do infrastructure, build or debugging work in a shell, and you need correctness decided rather than judged.
- Vigilance
- The score names a model and a harness together: the same weights under a weaker scaffold give up exactly where a stronger one reads the error and recovers.
Interrogez l'expert IA sur ce pattern
Ouvre l'assistant avec votre question préremplie. Vous la relisez avant l'envoi.
Terminal-Bench: Vue d’ensemble
Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.
- One container per task, so runs cannot contaminate each other
- Every task carries a human-written reference solution
- Verified by tests rather than by inspecting the transcript
- Drawn from real infrastructure, data and debugging work
- Deliberately hard: frontier systems remain well short of solving it
- Harness-agnostic, so the agent and the scaffold are measured together
The Agent Architect
Un pattern, un compromis, une panne de production racontée. Un brief hebdomadaire court pour ceux qui construisent des systèmes agentiques.
Un email par semaine, désinscription en un clic. Votre adresse ne sert qu'à envoyer le brief.
Aussi appelé : CLI agent benchmark, Command line agent evaluation
Références
Les articles, spécifications et dépôts sur lesquels repose ce modèle.