In the news
Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling
arXiv cs.AI · Published · 3 min read
In 30 seconds
- What happened
- Researchers propose ExecJudges, a method using LLMs to generate executable programs that replace per-document LLM judgments for relevance labeling in information retrieval evaluation.
- Why it matters
- Matters for engineers building retrieval systems or evaluation pipelines who need cost-effective, reproducible relevance labels at scale without repeated LLM API calls.
- Watch out
- Method relies on lexical matching and standard NLP tools, which may miss semantic nuances that frontier LLM judges capture. Generalization beyond TREC collections remains unclear.
- llm
- language model
- retrieval
- eval
The patterns behind this
- Eval-Driven Development (Agent CI)
- Query Transformation Retrieval
- Agentic Context Engineering (Evolving Playbook)
Each one covers how the technique works, when it earns its cost, and where it breaks.
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.