正在加载模式…
🏗️
SWE-bench Pro(SWE-Pro)
Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.
复杂度: high评估与监控
30秒速览
- 是什么
- Sets long-horizon software engineering tasks across 41 repositories chosen so that remembering the accepted patch is not a route to a good score.
- 何时使用
- SWE-bench numbers have saturated for the systems you are comparing, or contamination is the specific question being asked.
- 注意
- It moves the contamination horizon rather than removing it. This is still a fixed published set, and it will date the way its predecessor did.
向AI专家咨询此模式
打开助手并预填您的问题,发送前可先确认。
SWE-bench Pro: 概览
Software engineering tasks long enough to take a professional hours or days, drawn from repositories chosen so that memorising the answer is not an option.
- 1,865 problems across 41 actively maintained repositories
- Business, B2B and developer-tool codebases, not only popular OSS
- Patches typically span multiple files and substantial rewrites
- Held-out and commercially licensed subsets to resist contamination
- Human-verified problem statements and interfaces
- Built as the answer to SWE-bench saturation and leakage
The Agent Architect
每周一个模式、一个权衡、一个生产事故案例。为构建智能体系统的人准备的每周简报。
每周一封邮件,一键退订。您的地址仅用于发送简报。
又称： Long-horizon SWE benchmark, Enterprise coding benchmark
参考资料
该模式所依据的论文、规范和代码仓库。