Loading patterns…

🏗️

SWE-bench Pro(SWE-Pro)

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

Complexity: highEvaluation and Monitoring

In 30 seconds

What: Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
When to use: SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
Watch out: It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.

Ask the AI expert about this pattern

Opens the assistant with your question prefilled. You review it before sending.

Get this catalog as an MCP server

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

1,865 problems across 41 actively maintained repositories
Business, B2B and developer-tool codebases, not only popular OSS
Patches typically span multiple files and substantial rewrites
Held-out and commercially licensed subsets to resist contamination
Human-verified problem statements and interfaces
Built as the answer to SWE-bench saturation and leakage

The Agent Architect

One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.

Weekly email, one-click unsubscribe. We only use your address to send the briefing.

Also known as: Long-horizon SWE benchmark, Enterprise coding benchmark

References

The papers, specifications, and repositories this pattern is based on.

Patterns

Loading patterns…

Design Patterns & Techniques

Loading patterns…

🏗️

SWE-bench Pro(SWE-Pro)

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

Complexity: highEvaluation and Monitoring

In 30 seconds

What: Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
When to use: SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
Watch out: It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.

Ask the AI expert about this pattern

Opens the assistant with your question prefilled. You review it before sending.

Get this catalog as an MCP server

30-Second Overview

Pattern: Long-horizon software engineering tasks across repositories chosen so that remembering the patch is not a route to a good score

Why: SWE-bench saturated and its repositories sit in every pretraining corpus, so a high score stopped distinguishing reasoning from recall

Key Insight: It moves the contamination horizon rather than removing it: this is still a fixed published set, and it will date the same way

Quick Implementation

1Select:Choose the public or held-out subset, and state which one every reported number came from

2Provision:Build the repository environment at the pinned commit with its dependencies

3Instruct:Give the human-verified problem statement and interface, nothing more

4Budget:Allow a horizon that matches the task: these are hours of professional work, not minutes

5Verify:Run the repository test suite and confirm the patch breaks nothing that previously passed

Example: swe_pro = SWEBenchPro(subset="held_out", repos=41, agent=coding_agent, timeout_s=14400)

Do's & Don'ts

✅State the subset: public and held-out numbers are not comparable

✅Budget a horizon that matches the task, or you are measuring the cap

✅Check the patch does not break tests that already passed

✅Read it next to SWE-bench Verified: the gap between them is the contamination estimate

✅Record cost per resolved task, since long-horizon runs are expensive enough to change the decision

❌Quote a Pro score as if it were a SWE-bench score

❌Assume the held-out set stays uncontaminated indefinitely

❌Expect it to predict performance on your codebase, whose conventions are in neither set

❌Run it often enough to overfit the harness to 41 repositories

❌Read a low score as incapability without separating timeouts from wrong patches

When to Use

Use When

• SWE-bench Verified has saturated for the systems you are comparing
• Contamination is the specific question being asked
• The work you are automating spans files and takes hours rather than minutes
• You need a coding number that will still separate systems next year

Avoid When

• You need a fast or cheap signal, where SWE-bench Verified remains the practical choice
• The target is a single-function change, which this over-tests
• Your language or stack is outside the covered repositories
• You cannot afford the compute for hours-long runs at reasonable sample sizes

Key Metrics

% Resolved

Problems whose patch makes the repository test suite pass

Public vs Held-Out Gap

Difference between subsets, the closest available contamination estimate

Files Touched

Patch breadth against the reference, exposing under-scoped fixes

Regression Rate

Patches that pass the target test and break another

Cost per Resolved Task

Compute spend per success, which long horizons make decisive

Timeout Share

Runs that exhausted the budget rather than producing a wrong patch

Top Use Cases

Post-Saturation Comparison: separating systems that SWE-bench Verified no longer distinguishes

Contamination Estimation: reading the public and held-out gap as a leakage signal

Long-Horizon Capability: whether an agent holds a design across files for hours

Enterprise Codebase Proxy: business and B2B repositories rather than popular OSS only

Release Gating: a slow, expensive check reserved for model or harness changes

References & Further Reading

Deepen your understanding with these curated resources

Original Research

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv:2509.16941)

Context and Predecessor

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770)

The SWE-Bench Illusion: When LLMs Remember Instead of Reason (arXiv:2506.12286)

SWE-bench Pro repository

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

1,865 problems across 41 actively maintained repositories
Business, B2B and developer-tool codebases, not only popular OSS
Patches typically span multiple files and substantial rewrites
Held-out and commercially licensed subsets to resist contamination
Human-verified problem statements and interfaces
Built as the answer to SWE-bench saturation and leakage

The Agent Architect

One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.

Weekly email, one-click unsubscribe. We only use your address to send the briefing.

Loading patterns…

SWE-bench Pro(SWE-Pro)

In 30 seconds

Ask the AI expert about this pattern

SWE-bench Pro: Overview

The Agent Architect

References

Patterns

Loading patterns…

Design Patterns & Techniques

Loading patterns…

SWE-bench Pro(SWE-Pro)

In 30 seconds

Ask the AI expert about this pattern

30-Second Overview

Quick Implementation

Do's & Don'ts

When to Use

Use When

Avoid When

Key Metrics

Top Use Cases

References & Further Reading

Original Research

Context and Predecessor

SWE-bench Pro: Overview

The Agent Architect

References

Patterns

Design Patterns & Techniques

Loading patterns…

Design Patterns & Techniques

30-Second Overview

Quick Implementation

Do's & Don'ts

When to Use

Use When

Avoid When

Key Metrics

Top Use Cases

References & Further Reading

Original Research

Context and Predecessor

Design Patterns & Techniques