Chargement des modèles…

🏗️

SWE-bench Pro(SWE-Pro)

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

Complexité: highÉvaluation et surveillance

Aperçu en 30 secondes

Quoi: Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
Quand l'utiliser: SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
Vigilance: It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.

Interrogez l'expert IA sur ce pattern

Ouvre l'assistant avec votre question préremplie. Vous la relisez avant l'envoi.

Obtenir ce catalogue en serveur MCP

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

1,865 problems across 41 actively maintained repositories
Business, B2B and developer-tool codebases, not only popular OSS
Patches typically span multiple files and substantial rewrites
Held-out and commercially licensed subsets to resist contamination
Human-verified problem statements and interfaces
Built as the answer to SWE-bench saturation and leakage

The Agent Architect

Un pattern, un compromis, une panne de production racontée. Un brief hebdomadaire court pour ceux qui construisent des systèmes agentiques.

Un email par semaine, désinscription en un clic. Votre adresse ne sert qu'à envoyer le brief.

Aussi appelé : Long-horizon SWE benchmark, Enterprise coding benchmark

Références

Les articles, spécifications et dépôts sur lesquels repose ce modèle.

Modèles

Chargement des modèles…

Modèles et techniques de conception

Chargement des modèles…

🏗️

SWE-bench Pro(SWE-Pro)

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

Complexité: highÉvaluation et surveillance

Aperçu en 30 secondes

Quoi: Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
Quand l'utiliser: SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
Vigilance: It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.

Interrogez l'expert IA sur ce pattern

Ouvre l'assistant avec votre question préremplie. Vous la relisez avant l'envoi.

Obtenir ce catalogue en serveur MCP

Aperçu en 30 secondes

Modèle : Long-horizon software engineering tasks across repositories chosen so that remembering the patch is not a route to a good score

Pourquoi : SWE-bench saturated and its repositories sit in every pretraining corpus, so a high score stopped distinguishing reasoning from recall

Idée clé : It moves the contamination horizon rather than removing it: this is still a fixed published set, and it will date the same way

Mise en œuvre rapide

1Select:Choose the public or held-out subset, and state which one every reported number came from

2Provision:Build the repository environment at the pinned commit with its dependencies

3Instruct:Give the human-verified problem statement and interface, nothing more

4Budget:Allow a horizon that matches the task: these are hours of professional work, not minutes

5Verify:Run the repository test suite and confirm the patch breaks nothing that previously passed

Example: swe_pro = SWEBenchPro(subset="held_out", repos=41, agent=coding_agent, timeout_s=14400)

À faire et à éviter

✅State the subset: public and held-out numbers are not comparable

✅Budget a horizon that matches the task, or you are measuring the cap

✅Check the patch does not break tests that already passed

✅Read it next to SWE-bench Verified: the gap between them is the contamination estimate

✅Record cost per resolved task, since long-horizon runs are expensive enough to change the decision

❌Quote a Pro score as if it were a SWE-bench score

❌Assume the held-out set stays uncontaminated indefinitely

❌Expect it to predict performance on your codebase, whose conventions are in neither set

❌Run it often enough to overfit the harness to 41 repositories

❌Read a low score as incapability without separating timeouts from wrong patches

Quand l’utiliser

À utiliser quand

• SWE-bench Verified has saturated for the systems you are comparing
• Contamination is the specific question being asked
• The work you are automating spans files and takes hours rather than minutes
• You need a coding number that will still separate systems next year

À éviter quand

• You need a fast or cheap signal, where SWE-bench Verified remains the practical choice
• The target is a single-function change, which this over-tests
• Your language or stack is outside the covered repositories
• You cannot afford the compute for hours-long runs at reasonable sample sizes

Indicateurs clés

% Resolved

Problems whose patch makes the repository test suite pass

Public vs Held-Out Gap

Difference between subsets, the closest available contamination estimate

Files Touched

Patch breadth against the reference, exposing under-scoped fixes

Regression Rate

Patches that pass the target test and break another

Cost per Resolved Task

Compute spend per success, which long horizons make decisive

Timeout Share

Runs that exhausted the budget rather than producing a wrong patch

Principaux cas d’usage

Post-Saturation Comparison: separating systems that SWE-bench Verified no longer distinguishes

Contamination Estimation: reading the public and held-out gap as a leakage signal

Long-Horizon Capability: whether an agent holds a design across files for hours

Enterprise Codebase Proxy: business and B2B repositories rather than popular OSS only

Release Gating: a slow, expensive check reserved for model or harness changes

Références et lectures complémentaires

Approfondissez le sujet avec ces ressources sélectionnées

Original Research

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv:2509.16941)

Context and Predecessor

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770)

The SWE-Bench Illusion: When LLMs Remember Instead of Reason (arXiv:2506.12286)

SWE-bench Pro repository

Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.

1,865 problems across 41 actively maintained repositories
Business, B2B and developer-tool codebases, not only popular OSS
Patches typically span multiple files and substantial rewrites
Held-out and commercially licensed subsets to resist contamination
Human-verified problem statements and interfaces
Built as the answer to SWE-bench saturation and leakage

The Agent Architect

Un pattern, un compromis, une panne de production racontée. Un brief hebdomadaire court pour ceux qui construisent des systèmes agentiques.

Un email par semaine, désinscription en un clic. Votre adresse ne sert qu'à envoyer le brief.

Chargement des modèles…

SWE-bench Pro(SWE-Pro)

Aperçu en 30 secondes

Interrogez l'expert IA sur ce pattern

SWE-bench Pro: Vue d’ensemble

The Agent Architect

Références

Modèles

Chargement des modèles…

Modèles et techniques de conception

Chargement des modèles…

SWE-bench Pro(SWE-Pro)

Aperçu en 30 secondes

Interrogez l'expert IA sur ce pattern

Aperçu en 30 secondes

Mise en œuvre rapide

À faire et à éviter

Quand l’utiliser

À utiliser quand

À éviter quand

Indicateurs clés

Principaux cas d’usage

Références et lectures complémentaires

Original Research

Context and Predecessor

SWE-bench Pro: Vue d’ensemble

The Agent Architect

Références

Modèles

Modèles et techniques de conception

Chargement des modèles…

Modèles et techniques de conception

Aperçu en 30 secondes

Mise en œuvre rapide

À faire et à éviter

Quand l’utiliser

À utiliser quand

À éviter quand

Indicateurs clés

Principaux cas d’usage

Références et lectures complémentaires

Original Research

Context and Predecessor

Modèles et techniques de conception