Загружаем паттерны…

⌨️

Terminal-Bench(TB)

Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.

Сложность: highОценка и мониторинг

За 30 секунд

Что это: Poses hard command line tasks in isolated containers, each carrying a human-written solution and tests that decide afterwards whether the work was actually finished.
Когда применять: You are evaluating agents that do infrastructure, build or debugging work in a shell, and you need correctness decided rather than judged.
Осторожно: The score names a model and a harness together: the same weights under a weaker scaffold give up exactly where a stronger one reads the error and recovers.

Спросите ИИ-эксперта об этом паттерне

Откроет ассистента с готовым вопросом. Вы проверите его перед отправкой.

Получить этот каталог как MCP-сервер

Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.

One container per task, so runs cannot contaminate each other
Every task carries a human-written reference solution
Verified by tests rather than by inspecting the transcript
Drawn from real infrastructure, data and debugging work
Deliberately hard: frontier systems remain well short of solving it
Harness-agnostic, so the agent and the scaffold are measured together

The Agent Architect

Один паттерн, один компромисс, одна история сбоя в продакшене. Короткий еженедельный брифинг для тех, кто строит агентные системы.

Одно письмо в неделю, отписка в один клик. Адрес используется только для рассылки брифинга.

Также известен как: CLI agent benchmark, Command line agent evaluation

Источники

Статьи, спецификации и репозитории, на которых основан этот паттерн.

Паттерны

Загружаем паттерны…

Паттерны и методы проектирования

Загружаем паттерны…

⌨️

Terminal-Bench(TB)

Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.

Сложность: highОценка и мониторинг

За 30 секунд

Что это: Poses hard command line tasks in isolated containers, each carrying a human-written solution and tests that decide afterwards whether the work was actually finished.
Когда применять: You are evaluating agents that do infrastructure, build or debugging work in a shell, and you need correctness decided rather than judged.
Осторожно: The score names a model and a harness together: the same weights under a weaker scaffold give up exactly where a stronger one reads the error and recovers.

Спросите ИИ-эксперта об этом паттерне

Откроет ассистента с готовым вопросом. Вы проверите его перед отправкой.

Получить этот каталог как MCP-сервер

Обзор за 30 секунд

Паттерн: Hard command-line tasks in isolated containers, each with a human-written solution and tests that decide whether the work was finished

Почему: The terminal is where infrastructure and data work actually happens, and it is one of the few settings where correctness can be decided rather than judged

Ключевая идея: Because the interface is generic, the harness is measured alongside the model: the same model scores differently under two scaffolds

Быстрая реализация

1Pull:Fetch the task image so the environment is byte-identical to every other run of it

2Instruct:Give the task statement and a shell, with no pointer to the relevant files

3Run:Let the agent explore, edit and execute under a step and time budget

4Test:Run the task-provided suite in a fresh shell after the agent stops

5Attribute:Record the model and the harness together, since the pair is what produced the score

Example: terminal_bench = TerminalBench(task="fix-broken-build", agent=my_harness, timeout_s=900)

Что делать и чего избегать

✅Report the harness alongside the model, because the benchmark measures both

✅Run each task in its own container so no run can contaminate another

✅Keep the transcript: how an agent recovers from a failed command is the interesting part

✅Treat the reference solution as proof of achievability, never as a hint to the agent

✅Expect low absolute scores and read movement rather than level

❌Award partial credit: the suite passes or it does not, and softening that removes the signal

❌Compare a Terminal-Bench 2.0 number against a 1.0 number

❌Generalise a terminal score to agent work that cannot be asserted against

❌Let the agent reach the network unless the task intends it, which makes runs irreproducible

❌Read a timeout as incapability without checking whether it was looping

Когда использовать

Используйте, когда

• Evaluating agents that do infrastructure, build, data or debugging work in a shell
• Comparing agent harnesses on a fixed model, where the scaffold is the variable
• You need decidable correctness rather than a graded judgement
• Reproducibility matters more than breadth of task type

Избегайте, когда

• The target work is graphical, where OSWorld is the right environment
• You need many trials cheaply: these tasks are deliberately long
• The question is knowledge rather than execution
• Your agent has no shell access in production anyway

Ключевые метрики

Resolution Rate

Fraction of tasks whose provided test suite passes afterwards

Harness Delta

Score difference between two scaffolds running the identical model

Recovery Rate

Runs that reached a passing state after an early failed command

Commands to Completion

Actions used against the human reference solution

Timeout Share

Runs that hit the budget, separating looping from difficulty

Destructive Actions

Runs that damaged the environment beyond recovery

Основные сценарии использования

CLI Agent Evaluation: the closest available proxy for real terminal work

Harness Comparison: isolating scaffold quality by holding the model fixed

Regression Gates: a reproducible suite that can block a release

Recovery Analysis: studying what an agent does after a command that did nothing

DevOps Automation Readiness: evidence before granting shell access in anger

Источники и дополнительные материалы

Углубите понимание с помощью этих подобранных материалов

Original Research

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (arXiv:2601.11868)

Project

Terminal-Bench project site

Hard command-line tasks in isolated environments, each with a human-written solution and tests that decide whether the agent actually finished.

One container per task, so runs cannot contaminate each other
Every task carries a human-written reference solution
Verified by tests rather than by inspecting the transcript
Drawn from real infrastructure, data and debugging work
Deliberately hard: frontier systems remain well short of solving it
Harness-agnostic, so the agent and the scaffold are measured together

The Agent Architect

Одно письмо в неделю, отписка в один клик. Адрес используется только для рассылки брифинга.

Также известен как: CLI agent benchmark, Command line agent evaluation

Источники

Статьи, спецификации и репозитории, на которых основан этот паттерн.

Загружаем паттерны…

Terminal-Bench(TB)

За 30 секунд

Спросите ИИ-эксперта об этом паттерне

Terminal-Bench: Обзор

The Agent Architect

Источники

Паттерны

Загружаем паттерны…

Паттерны и методы проектирования

Загружаем паттерны…

Terminal-Bench(TB)

За 30 секунд

Спросите ИИ-эксперта об этом паттерне

Обзор за 30 секунд

Быстрая реализация

Что делать и чего избегать

Когда использовать

Используйте, когда

Избегайте, когда

Ключевые метрики

Основные сценарии использования

Источники и дополнительные материалы

Original Research

Project

Terminal-Bench: Обзор

The Agent Architect

Источники

Паттерны

Паттерны и методы проектирования

Загружаем паттерны…

Паттерны и методы проектирования

Обзор за 30 секунд

Быстрая реализация

Что делать и чего избегать

Когда использовать

Используйте, когда

Избегайте, когда

Ключевые метрики

Основные сценарии использования

Источники и дополнительные материалы

Original Research

Project

Паттерны и методы проектирования