Loading patterns…
tau-bench (Tool-Agent-User)(TAU)
Benchmark that places the agent between a simulated human user and a set of domain APIs (retail, airline) while it must obey a written policy document. Rather than checking a single response, it compares the final database state against a goal state after the full conversation. Its signature metric is pass^k, the probability of succeeding on all k independent trials of the same task, which exposes consistency failures that averaged pass@1 scores hide.
In 30 seconds
- What
- Evaluates agents by simulating a user-agent-API conversation, scoring final database state against goal state across multiple independent trials to measure consistency.
- When to use
- Testing whether agents reliably follow domain policies under user pressure while manipulating real system state across retail, airline, or telecom workflows.
- Watch out
- High variance in pass^k scores can mask that an agent succeeds once by luck but fails systematically; requires many trials to detect real brittleness.
Ask the AI expert about this pattern
Opens the assistant with your question prefilled. You review it before sending.
tau-bench (Tool-Agent-User): Overview
Benchmark that places the agent between a simulated human user and a set of domain APIs (retail, airline) while it must obey a written policy document. Rather than checking a single response, it compares the final database state against a goal state after the full conversation. Its signature metric is pass^k, the probability of succeeding on all k independent trials of the same task, which exposes consistency failures that averaged pass@1 scores hide.
- Agent mediates between a simulated user (LLM-driven) and domain APIs
- Two domains: retail and airline, each with tools and a written policy document
- Scores the final database state against a goal state, not just the text reply
- pass^k reliability metric across k independent trials of the same task
- Policy adherence testing: agent must follow domain rules under user pressure
- tau2-bench extends the setting with a dual-control telecom domain and stronger user simulation
Get the Agent Evals field guide
All 25 agent evaluation methods condensed into one guide: which benchmark measures what, when a public score misleads you, and how to build evals out of your own failures. The link arrives with your confirmation, alongside the weekly Agent Architect.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.
References
The papers, specifications, and repositories this pattern is based on.
From the engineer behind this catalog
Find out what your evals miss
Measuring an agent is harder than shipping one, and most suites stay green while production drifts. Have your evaluation setup reviewed end to end: what you measure today, what you cannot see yet, and the regressions your current suite would let through.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September