Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

In 30 Sekunden

Was passiert ist

Researchers propose treating benchmarks as living software requiring continuous maintenance, releasing Terminal-Bench 2.1 with fixes addressing 30% of prior version issues.

Warum es zählt

Engineers evaluating coding agents or LLM systems need this when benchmark scores diverge from actual model capabilities or when infrastructure changes invalidate test assumptions.

Achtung

Repairs changed accuracy by 10+ points and leaderboard rankings by up to three positions, showing benchmark maintenance directly affects performance claims and model comparisons.