Muster werden geladen…

🏗️

SWE-bench Pro(SWE-Pro)

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

Komplexität: highEvaluierung und Monitoring

In 30 Sekunden

Was: Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
Wann einsetzen: SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
Achtung: It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.

Fragen Sie den KI-Experten zu diesem Pattern

Öffnet den Assistenten mit vorbereiteter Frage. Sie prüfen sie vor dem Senden.

Diesen Katalog als MCP-Server erhalten

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

1,865 problems across 41 actively maintained repositories
Business, B2B and developer-tool codebases, not only popular OSS
Patches typically span multiple files and substantial rewrites
Held-out and commercially licensed subsets to resist contamination
Human-verified problem statements and interfaces
Built as the answer to SWE-bench saturation and leakage

The Agent Architect

Ein Pattern, ein Tradeoff, eine Produktionspanne. Ein kurzes wöchentliches Briefing für alle, die agentische Systeme bauen.

Wöchentliche E-Mail, Abmeldung mit einem Klick. Ihre Adresse wird nur für das Briefing verwendet.

Auch bekannt als: Long-horizon SWE benchmark, Enterprise coding benchmark

Quellen

Die Veröffentlichungen, Spezifikationen und Repositories, auf denen dieses Muster beruht.

Muster

Muster werden geladen…

Entwurfsmuster und Techniken

Muster werden geladen…

🏗️

SWE-bench Pro(SWE-Pro)

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

Komplexität: highEvaluierung und Monitoring

In 30 Sekunden

Was: Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
Wann einsetzen: SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
Achtung: It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.

Fragen Sie den KI-Experten zu diesem Pattern

Öffnet den Assistenten mit vorbereiteter Frage. Sie prüfen sie vor dem Senden.

Diesen Katalog als MCP-Server erhalten

30-Sekunden-Überblick

Muster: Long-horizon software engineering tasks across repositories chosen so that remembering the patch is not a route to a good score

Warum: SWE-bench saturated and its repositories sit in every pretraining corpus, so a high score stopped distinguishing reasoning from recall

Kernerkenntnis: It moves the contamination horizon rather than removing it: this is still a fixed published set, and it will date the same way

Schnelle Umsetzung

1Select:Choose the public or held-out subset, and state which one every reported number came from

2Provision:Build the repository environment at the pinned commit with its dependencies

3Instruct:Give the human-verified problem statement and interface, nothing more

4Budget:Allow a horizon that matches the task: these are hours of professional work, not minutes

5Verify:Run the repository test suite and confirm the patch breaks nothing that previously passed

Example: swe_pro = SWEBenchPro(subset="held_out", repos=41, agent=coding_agent, timeout_s=14400)

Do’s und Don’ts

✅State the subset: public and held-out numbers are not comparable

✅Budget a horizon that matches the task, or you are measuring the cap

✅Check the patch does not break tests that already passed

✅Read it next to SWE-bench Verified: the gap between them is the contamination estimate

✅Record cost per resolved task, since long-horizon runs are expensive enough to change the decision

❌Quote a Pro score as if it were a SWE-bench score

❌Assume the held-out set stays uncontaminated indefinitely

❌Expect it to predict performance on your codebase, whose conventions are in neither set

❌Run it often enough to overfit the harness to 41 repositories

❌Read a low score as incapability without separating timeouts from wrong patches

Wann einsetzen

Einsetzen, wenn

• SWE-bench Verified has saturated for the systems you are comparing
• Contamination is the specific question being asked
• The work you are automating spans files and takes hours rather than minutes
• You need a coding number that will still separate systems next year

Vermeiden, wenn

• You need a fast or cheap signal, where SWE-bench Verified remains the practical choice
• The target is a single-function change, which this over-tests
• Your language or stack is outside the covered repositories
• You cannot afford the compute for hours-long runs at reasonable sample sizes

Kennzahlen

% Resolved

Problems whose patch makes the repository test suite pass

Public vs Held-Out Gap

Difference between subsets, the closest available contamination estimate

Files Touched

Patch breadth against the reference, exposing under-scoped fixes

Regression Rate

Patches that pass the target test and break another

Cost per Resolved Task

Compute spend per success, which long horizons make decisive

Timeout Share

Runs that exhausted the budget rather than producing a wrong patch

Wichtigste Anwendungsfälle

Post-Saturation Comparison: separating systems that SWE-bench Verified no longer distinguishes

Contamination Estimation: reading the public and held-out gap as a leakage signal

Long-Horizon Capability: whether an agent holds a design across files for hours

Enterprise Codebase Proxy: business and B2B repositories rather than popular OSS only

Release Gating: a slow, expensive check reserved for model or harness changes

Referenzen und weiterführende Literatur

Vertiefen Sie Ihr Verständnis mit diesen ausgewählten Ressourcen

Original Research

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv:2509.16941)

Context and Predecessor

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770)

The SWE-Bench Illusion: When LLMs Remember Instead of Reason (arXiv:2506.12286)

SWE-bench Pro repository

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

1,865 problems across 41 actively maintained repositories
Business, B2B and developer-tool codebases, not only popular OSS
Patches typically span multiple files and substantial rewrites
Held-out and commercially licensed subsets to resist contamination
Human-verified problem statements and interfaces
Built as the answer to SWE-bench saturation and leakage

The Agent Architect

Ein Pattern, ein Tradeoff, eine Produktionspanne. Ein kurzes wöchentliches Briefing für alle, die agentische Systeme bauen.

Wöchentliche E-Mail, Abmeldung mit einem Klick. Ihre Adresse wird nur für das Briefing verwendet.

Auch bekannt als: Long-horizon SWE benchmark, Enterprise coding benchmark

Quellen

Die Veröffentlichungen, Spezifikationen und Repositories, auf denen dieses Muster beruht.

Muster werden geladen…

SWE-bench Pro(SWE-Pro)

In 30 Sekunden

Fragen Sie den KI-Experten zu diesem Pattern

SWE-bench Pro: Überblick

The Agent Architect

Quellen

Muster

Muster werden geladen…

Entwurfsmuster und Techniken

Muster werden geladen…

SWE-bench Pro(SWE-Pro)

In 30 Sekunden

Fragen Sie den KI-Experten zu diesem Pattern

30-Sekunden-Überblick

Schnelle Umsetzung

Do’s und Don’ts

Wann einsetzen

Einsetzen, wenn

Vermeiden, wenn

Kennzahlen

Wichtigste Anwendungsfälle

Referenzen und weiterführende Literatur

Original Research

Context and Predecessor

SWE-bench Pro: Überblick

The Agent Architect

Quellen

Muster

Entwurfsmuster und Techniken

Muster werden geladen…

Entwurfsmuster und Techniken

30-Sekunden-Überblick

Schnelle Umsetzung

Do’s und Don’ts

Wann einsetzen

Einsetzen, wenn

Vermeiden, wenn

Kennzahlen

Wichtigste Anwendungsfälle

Referenzen und weiterführende Literatur

Original Research

Context and Predecessor

Entwurfsmuster und Techniken