PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

In 30 Sekunden

Was passiert ist

Researchers found 13.6% of SWE-bench instances have misaligned PR-Issue pairs and built PAIChecker, a multi-agent system to detect such misalignment.

Warum es zählt

Matters for engineers building or using LLM benchmarks for code tasks, where incorrect problem-solution pairings corrupt evaluation results.

Achtung

The study covers only SWE-bench Verified instances; generalization to other benchmarks or real-world repositories remains unclear.