Загружаем паттерны…

🏗️

SWE-bench Pro(SWE-Pro)

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

Сложность: highОценка и мониторинг

За 30 секунд

Что это: Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
Когда применять: SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
Осторожно: It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.

Спросите ИИ-эксперта об этом паттерне

Откроет ассистента с готовым вопросом. Вы проверите его перед отправкой.

Получить этот каталог как MCP-сервер

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

1,865 problems across 41 actively maintained repositories
Business, B2B and developer-tool codebases, not only popular OSS
Patches typically span multiple files and substantial rewrites
Held-out and commercially licensed subsets to resist contamination
Human-verified problem statements and interfaces
Built as the answer to SWE-bench saturation and leakage

The Agent Architect

Один паттерн, один компромисс, одна история сбоя в продакшене. Короткий еженедельный брифинг для тех, кто строит агентные системы.

Одно письмо в неделю, отписка в один клик. Адрес используется только для рассылки брифинга.

Также известен как: Long-horizon SWE benchmark, Enterprise coding benchmark

Источники

Статьи, спецификации и репозитории, на которых основан этот паттерн.

Паттерны

Загружаем паттерны…

Паттерны и методы проектирования

Загружаем паттерны…

🏗️

SWE-bench Pro(SWE-Pro)

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

Сложность: highОценка и мониторинг

За 30 секунд

Что это: Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
Когда применять: SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
Осторожно: It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.

Спросите ИИ-эксперта об этом паттерне

Откроет ассистента с готовым вопросом. Вы проверите его перед отправкой.

Получить этот каталог как MCP-сервер

Обзор за 30 секунд

Паттерн: Long-horizon software engineering tasks across repositories chosen so that remembering the patch is not a route to a good score

Почему: SWE-bench saturated and its repositories sit in every pretraining corpus, so a high score stopped distinguishing reasoning from recall

Ключевая идея: It moves the contamination horizon rather than removing it: this is still a fixed published set, and it will date the same way

Быстрая реализация

1Select:Choose the public or held-out subset, and state which one every reported number came from

2Provision:Build the repository environment at the pinned commit with its dependencies

3Instruct:Give the human-verified problem statement and interface, nothing more

4Budget:Allow a horizon that matches the task: these are hours of professional work, not minutes

5Verify:Run the repository test suite and confirm the patch breaks nothing that previously passed

Example: swe_pro = SWEBenchPro(subset="held_out", repos=41, agent=coding_agent, timeout_s=14400)

Что делать и чего избегать

✅State the subset: public and held-out numbers are not comparable

✅Budget a horizon that matches the task, or you are measuring the cap

✅Check the patch does not break tests that already passed

✅Read it next to SWE-bench Verified: the gap between them is the contamination estimate

✅Record cost per resolved task, since long-horizon runs are expensive enough to change the decision

❌Quote a Pro score as if it were a SWE-bench score

❌Assume the held-out set stays uncontaminated indefinitely

❌Expect it to predict performance on your codebase, whose conventions are in neither set

❌Run it often enough to overfit the harness to 41 repositories

❌Read a low score as incapability without separating timeouts from wrong patches

Когда использовать

Используйте, когда

• SWE-bench Verified has saturated for the systems you are comparing
• Contamination is the specific question being asked
• The work you are automating spans files and takes hours rather than minutes
• You need a coding number that will still separate systems next year

Избегайте, когда

• You need a fast or cheap signal, where SWE-bench Verified remains the practical choice
• The target is a single-function change, which this over-tests
• Your language or stack is outside the covered repositories
• You cannot afford the compute for hours-long runs at reasonable sample sizes

Ключевые метрики

% Resolved

Problems whose patch makes the repository test suite pass

Public vs Held-Out Gap

Difference between subsets, the closest available contamination estimate

Files Touched

Patch breadth against the reference, exposing under-scoped fixes

Regression Rate

Patches that pass the target test and break another

Cost per Resolved Task

Compute spend per success, which long horizons make decisive

Timeout Share

Runs that exhausted the budget rather than producing a wrong patch

Основные сценарии использования

Post-Saturation Comparison: separating systems that SWE-bench Verified no longer distinguishes

Contamination Estimation: reading the public and held-out gap as a leakage signal

Long-Horizon Capability: whether an agent holds a design across files for hours

Enterprise Codebase Proxy: business and B2B repositories rather than popular OSS only

Release Gating: a slow, expensive check reserved for model or harness changes

Источники и дополнительные материалы

Углубите понимание с помощью этих подобранных материалов

Original Research

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv:2509.16941)

Context and Predecessor

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770)

The SWE-Bench Illusion: When LLMs Remember Instead of Reason (arXiv:2506.12286)

SWE-bench Pro repository

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

1,865 problems across 41 actively maintained repositories
Business, B2B and developer-tool codebases, not only popular OSS
Patches typically span multiple files and substantial rewrites
Held-out and commercially licensed subsets to resist contamination
Human-verified problem statements and interfaces
Built as the answer to SWE-bench saturation and leakage

The Agent Architect

Одно письмо в неделю, отписка в один клик. Адрес используется только для рассылки брифинга.

Также известен как: Long-horizon SWE benchmark, Enterprise coding benchmark

Источники

Статьи, спецификации и репозитории, на которых основан этот паттерн.

Загружаем паттерны…

SWE-bench Pro(SWE-Pro)

За 30 секунд

Спросите ИИ-эксперта об этом паттерне

SWE-bench Pro: Обзор

The Agent Architect

Источники

Паттерны

Загружаем паттерны…

Паттерны и методы проектирования

Загружаем паттерны…

SWE-bench Pro(SWE-Pro)

За 30 секунд

Спросите ИИ-эксперта об этом паттерне

Обзор за 30 секунд

Быстрая реализация

Что делать и чего избегать

Когда использовать

Используйте, когда

Избегайте, когда

Ключевые метрики

Основные сценарии использования

Источники и дополнительные материалы

Original Research

Context and Predecessor

SWE-bench Pro: Обзор

The Agent Architect

Источники

Паттерны

Паттерны и методы проектирования

Загружаем паттерны…

Паттерны и методы проектирования

Обзор за 30 секунд

Быстрая реализация

Что делать и чего избегать

Когда использовать

Используйте, когда

Избегайте, когда

Ключевые метрики

Основные сценарии использования

Источники и дополнительные материалы

Original Research

Context and Predecessor

Паттерны и методы проектирования