Agent benchmarks compared

15 benchmarks, grouped by the question they answer rather than by who published them. Each one is followed by the thing its own paper is quietest about: what a good score still does not prove, and the year it was published, because a good part of choosing between them is noticing which ones are answering a question from three model generations ago.

There is no ranking here, and no scores. Leaderboard numbers go stale in weeks and every one of these benchmarks is measuring a different thing, so a table of them side by side would be a table of numbers that cannot be compared. The useful question is which one is asking yours.

General capability

Can this agent handle varied tasks at all?

AgentBench2023

8 distinct environments, multi-turn.

Breadth: whether one agent holds up across unrelated environments.

Reach for it when: You are choosing between models and want one number that is not domain-specific.
What a good score does not prove: Breadth is not depth. An agent can place well across eight environments and still fail the one workflow you are shipping, because none of the eight is yours.

AgentBench: Evaluating LLMs as Agents (Liu et al., ICLR 2024)arXiv:2308.03688

GAIA: General AI Assistants Benchmark2023

450+ questions across three difficulty levels.

Whether reasoning, browsing and tool use combine on questions with one right answer.

Reach for it when: Your agent has to find a fact in the world rather than produce prose about it.
What a good score does not prove: Every question has an unambiguous answer, which is what makes it scorable and also what makes it unlike production, where the hard part is usually that the question is underspecified.

GAIA: a benchmark for General AI AssistantsarXiv:2311.12983

MMAU: Massive Multitask Agent Understanding2024

20 task designs, 3K+ prompts, five domains.

Capability split by domain and by skill, so a weak area is visible rather than averaged away.

Reach for it when: A single composite score has stopped telling you where a regression came from.
What a good score does not prove: It runs offline with no environment to act on, so it measures what a model knows how to do, not whether it survives contact with a real system.

MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse DomainsarXiv:2407.18961

Software engineering

Can it change a real codebase?

SWE-bench Suite2023

Real issues from 12+ repositories, with a human-validated subset.

Whether an agent resolves a real GitHub issue so the repository tests pass.

Reach for it when: You are building a coding agent and want the number the field will ask you for.
What a good score does not prove: The repositories are public and popular, which is exactly what makes contamination a live question and makes the score least transferable to a private codebase. SWE-bench Pro below exists because of it.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (ICLR 2024)arXiv:2310.06770

SWE-bench Pro2025

1,865 problems from 41 actively maintained repositories.

The same idea as SWE-bench against work that takes a professional hours or days, across repositories chosen to resist memorisation.

Reach for it when: SWE-bench numbers have saturated for you, or the contamination question is the one being asked.
What a good score does not prove: Harder and cleaner is still a fixed set. It moves the contamination horizon rather than removing it, and a benchmark this expensive to run gets run rarely enough that the number you are quoting may be several models old.

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (2025)arXiv:2509.16941

METR RE-Bench

Expert-level ML engineering tasks with a human baseline.

Research engineering work, scored against what human experts achieved on the same tasks.

Reach for it when: You need to say how close a frontier model is to a competent engineer, not to another model.
What a good score does not prove: A human baseline is a comparison to the humans who were measured, under a time limit. It is evidence about a task, not a claim about a profession.

Computer use

Can it operate the machine itself?

OSWorld2024

108 long-horizon workflows in 2.0, after 369 shorter tasks in the original.

Whether an agent completes real work in a real desktop environment, across applications, files and the OS underneath them.

Reach for it when: The agent is expected to use software a human uses, through the same interface a human gets.
What a good score does not prove: It scores the end state of a workflow, so an agent can reach it by a route no user would take and no operator would sanction. Getting there is not the same as being trusted to.

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (2024)arXiv:2404.07972

Terminal-Bench2026

89 curated tasks in 2.0, each with its own environment and solution.

The same question as OSWorld with the graphics removed: real command line work, verified by tests rather than by a screenshot.

Reach for it when: Your agent works where the work actually happens for infrastructure and data tasks, which is a shell.
What a good score does not prove: A terminal is unusually verifiable, which is what makes it a good benchmark and a poor sample of agent work in general. Nothing here tests an interface that cannot be asserted against.

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026)arXiv:2601.11868

Web interaction

Can it drive a browser to a goal?

WebArena Evaluation Suite2023

Reproducible sandboxed sites, plus visual and enterprise variants.

Whether multi-step browser work reaches the goal state, in a sandbox that replays identically.

Reach for it when: You need a web-agent number you can re-run and defend months later.
What a good score does not prove: Reproducibility is bought by freezing the web. Live sites change layout, fight bots and fail intermittently, and none of that is in the score.

WebArena: A Realistic Web Environment for Building Autonomous Agents (2023)arXiv:2307.13854

Tools, policy and state

Does it follow the rules while changing real records?

TheAgentCompany Benchmark2024

Multi-step tasks across simulated company roles.

Long professional tasks spanning several job roles, with credit for partial progress.

Reach for it when: The work you are automating takes hours and crosses functions.
What a good score does not prove: Partial credit is the point and also the trap: a respectable score can be assembled entirely from progress that stopped short of anything shippable.

TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (2024)arXiv:2412.14161

tau-bench (Tool-Agent-User)2024

Retail and airline domains, each with tools and a policy document.

Whether an agent follows a written policy while a simulated user pushes back, scored on the database it leaves behind.

Reach for it when: Being right once is not enough because the agent is touching real records.
What a good score does not prove: Its pass^k metric is the one thing here that measures consistency rather than peak. That also makes it the benchmark most likely to disagree with the others about the same agent.

tau-bench: A Benchmark for Tool-Agent-User Interaction (Sierra, arXiv:2406.12045)arXiv:2406.12045

Research work

Can it do open-ended technical work?

MLR-Bench2025

201 tasks across 9 research areas.

Open-ended ML research tasks taken from real conference workshops.

Reach for it when: The output you want is a method and a write-up, not a passing test.
What a good score does not prove: The tasks come from published work, so they have known-good answers. Research that has already been done is the easiest kind to appear to do.

MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (2025)arXiv:2505.19955

Safety and security

What happens when someone tries to make it misbehave?

MLCommons AI Safety Benchmark v1.02024

12 hazard categories, multi-language.

Response safety across a fixed set of hazard categories, scored for the record.

Reach for it when: You need documented safety evidence a reviewer will accept.
What a good score does not prove: It measures the hazards on the list. Whoever attacks your system is specifically looking for the ones that are not.

Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? (NeurIPS 2024)arXiv:2407.21792

CybersecEval 3

Agent and multi-agent security evaluations.

Security posture of autonomous and multi-agent systems.

Reach for it when: The agent can reach systems where a compromise costs more than a bad answer.
What a good score does not prove: A benchmark is a fixed set of known attacks and the threat landscape is not fixed. Passing dates faster than it reassures, and it is not a substitute for red-teaming.

MAPS: Multilingual Agent Performance & Security2025

805 tasks across 12 languages.

Whether capability and safety hold up outside English.

Reach for it when: You ship to users who do not prompt in English.
What a good score does not prove: Twelve languages is twelve. Degradation in a thirteenth is not measured, and the languages with the least training data are the ones least likely to be on any list.

MAPS: A Multilingual Benchmark for Agent Performance and SecurityarXiv:2505.15935

If you only run one

Run the one whose failure mode you cannot afford. A coding agent that is wrong is annoying, so SWE-bench is a capability question. An agent with write access to a booking system that is wrong once in ten runs is an incident, which is why tau-bench measures consistency rather than peak. Most teams reach for the capability number first and discover the consistency one in production.

Every benchmark here has its own page with the mechanism, a runnable demo and the paper, in evaluation and monitoring.