In the news
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
arXiv cs.AI · Published · 3 min read
In 30 seconds
- What happened
- Researchers propose treating benchmarks as living software requiring continuous maintenance, releasing Terminal-Bench 2.1 with fixes addressing 30% of prior version issues.
- Why it matters
- Engineers evaluating coding agents or LLM systems need this when benchmark scores diverge from actual model capabilities or when infrastructure changes invalidate test assumptions.
- Watch out
- Repairs changed accuracy by 10+ points and leaderboard rankings by up to three positions, showing benchmark maintenance directly affects performance claims and model comparisons.
- agent
- llm
- eval
- benchmark
The patterns behind this
Each one covers how the technique works, when it earns its cost, and where it breaks.
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.