PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

За 30 секунд

Что произошло

Researchers found 13.6% of SWE-bench instances have misaligned PR-Issue pairs and built PAIChecker, a multi-agent system to detect such misalignment.

Почему это важно

Matters for engineers building or using LLM benchmarks for code tasks, where incorrect problem-solution pairings corrupt evaluation results.

На что обратить внимание

The study covers only SWE-bench Verified instances; generalization to other benchmarks or real-world repositories remains unclear.