Loading...
tau-bench (Tool-Agent-User)(TAU)
Benchmark that places the agent between a simulated human user and a set of domain APIs (retail, airline) while it must obey a written policy document. Rather than checking a single response, it compares the final database state against a goal state after the full conversation. Its signature metric is pass^k, the probability of succeeding on all k independent trials of the same task, which exposes consistency failures that averaged pass@1 scores hide.
Loading technique guide…
tau-bench (Tool-Agent-User)(TAU)
Benchmark that places the agent between a simulated human user and a set of domain APIs (retail, airline) while it must obey a written policy document. Rather than checking a single response, it compares the final database state against a goal state after the full conversation. Its signature metric is pass^k, the probability of succeeding on all k independent trials of the same task, which exposes consistency failures that averaged pass@1 scores hide.
Loading technique guide…