Agentic DesignPatterns and tools for production AI agents

Loading...

🔬

METR RE-Bench(RE-Bench)

Benchmark for measuring performance of frontier model agents on ML research engineering tasks, comparing against human expert capabilities.

Complexity: highEvaluation and Monitoring

In 30 seconds

What: Measures frontier model agent performance on ML research engineering tasks against human expert benchmarks across completion rate, code quality, and research insights.
When to use: Assessing whether frontier models can handle real research engineering work or comparing capability gains across model versions.
Watch out: Task selection bias heavily influences results; tasks too narrow or too similar to training data inflate agent scores relative to real-world research variety.

Loading technique guide…

Patterns

Loading...

Loading Agentic Design…

Agentic DesignPatterns and tools for production AI agents

Expert help Pricing

Design Patterns & Techniques

Loading...

🔬

METR RE-Bench(RE-Bench)

Benchmark for measuring performance of frontier model agents on ML research engineering tasks, comparing against human expert capabilities.

Complexity: highEvaluation and Monitoring

In 30 seconds

What: Measures frontier model agent performance on ML research engineering tasks against human expert benchmarks across completion rate, code quality, and research insights.
When to use: Assessing whether frontier models can handle real research engineering work or comparing capability gains across model versions.
Watch out: Task selection bias heavily influences results; tasks too narrow or too similar to training data inflate agent scores relative to real-world research variety.

30-Second Overview

Pattern: METR's benchmark comparing frontier AI agents against 71 human experts across 7 ML research engineering environments

Why: Evaluates R&D automation capabilities highlighted as key risk in frontier AI safety policies

Key Insight: Agents achieve 4x human performance at 2h budget but humans outperform 2x at 32h - time scaling matters

Quick Implementation

1Setup:Deploy 7 ML research engineering environments with GPU access

2Configure:Set time budgets (2h, 8h, 32h) and scoring functions

3Evaluate:Run agents on scaling laws, GPU kernels, and ML optimization tasks

4Compare:Benchmark against 71 human expert attempts (61 distinct experts)

5Analyze:Assess R&D automation capabilities vs human performance

Example: re_bench = REBench(environments=7, experts=71, models=[claude_3.5, o1_preview], budget="8h")

Do's & Don'ts

✅Test across all 7 environments for comprehensive R&D capability assessment

✅Compare performance at multiple time budgets (2h, 8h, 32h)

✅Focus on open-ended research engineering tasks, not classical ML

✅Provide GPU access for kernel optimization and scaling experiments

✅Use human expert baselines from 61 distinct researchers

❌Expect agents to outperform humans on extended time budgets (32h)

❌Focus only on short-term performance - humans show better scaling

❌Use publicly available solutions - benchmark designed to avoid this

❌Skip safety considerations for R&D automation capabilities

❌Ignore the 10x speed advantage agents have in solution generation

When to Use

Use When

• Evaluating frontier AI R&D automation capabilities
• Measuring research engineering vs classical ML skills
• Assessing AI safety risks from autonomous R&D
• Comparing agent performance against human experts
• Research on AI-driven scientific discovery

Avoid When

• Standard ML benchmarking with public solutions
• Classical machine learning task evaluation
• Short-term capability assessment only
• Environments without GPU/compute resources
• Non-research engineering skill evaluation

Key Metrics

Expert Success Rate

82% of human experts achieved non-zero scores

Strong Solution Match

24% of experts matched or exceeded reference solutions

Short-term Agent Advantage

4x higher scores than humans at 2-hour budget

Long-term Human Advantage

2x human scores vs top agents at 32-hour budget

Solution Generation Speed

Agents generate/test solutions 10x faster than humans

Cost Efficiency

Much lower cost per solution attempt for AI agents

Top Use Cases

AI Safety Research: Evaluating R&D automation risks as highlighted in frontier AI safety policies

Research Capability Assessment: Measuring agent performance on scaling laws, GPU kernel optimization

Human-AI Comparison: Direct benchmarking against 71 expert attempts across multiple time budgets

Autonomous R&D Evaluation: Reproducing the published roster (Claude 3.5 Sonnet, o1-preview) on research tasks

Policy and Governance: Supporting White House NSM and EU AI Act evaluations for R&D capabilities

References & Further Reading

Deepen your understanding with these curated resources

Official METR RE-Bench Resources

RE-Bench: Evaluating Frontier AI R&D Capabilities (arXiv:2411.15114)

METR Blog: Evaluating R&D Capabilities of LLMs

METR AI R&D Evaluation Report (PDF)

RE-Bench ResearchGate Publication

Conference & Academic Recognition

ICML 2025 Poster: RE-Bench

OpenReview: RE-Bench Discussion

Hugging Face Paper Page

Hacker News Discussion: RE-Bench

Related METR Research

METR: Measuring Autonomous AI Capabilities

KernelBench: Can LLMs Write Efficient GPU Kernels?

METR Blog: Measuring Automated Kernel Engineering

Apollo Research: Forecasting Frontier Agent Capabilities

Policy & Safety Context

EU Artificial Intelligence Act

NVIDIA: GPU Kernel Generation with DeepSeek-R1

LessWrong: Forecasting Frontier Agent Capabilities

Loading technique guide…

Patterns

Design Patterns & Techniques

Loading...

30-Second Overview

Pattern: METR's benchmark comparing frontier AI agents against 71 human experts across 7 ML research engineering environments

Why: Evaluates R&D automation capabilities highlighted as key risk in frontier AI safety policies

Key Insight: Agents achieve 4x human performance at 2h budget but humans outperform 2x at 32h - time scaling matters

Quick Implementation

1Setup:Deploy 7 ML research engineering environments with GPU access

2Configure:Set time budgets (2h, 8h, 32h) and scoring functions

3Evaluate:Run agents on scaling laws, GPU kernels, and ML optimization tasks

4Compare:Benchmark against 71 human expert attempts (61 distinct experts)

5Analyze:Assess R&D automation capabilities vs human performance

Example: re_bench = REBench(environments=7, experts=71, models=[claude_3.5, o1_preview], budget="8h")

Do's & Don'ts

✅Test across all 7 environments for comprehensive R&D capability assessment

✅Compare performance at multiple time budgets (2h, 8h, 32h)

✅Focus on open-ended research engineering tasks, not classical ML

✅Provide GPU access for kernel optimization and scaling experiments

✅Use human expert baselines from 61 distinct researchers

❌Expect agents to outperform humans on extended time budgets (32h)

❌Focus only on short-term performance - humans show better scaling

❌Use publicly available solutions - benchmark designed to avoid this

❌Skip safety considerations for R&D automation capabilities

❌Ignore the 10x speed advantage agents have in solution generation

When to Use

Use When

• Evaluating frontier AI R&D automation capabilities
• Measuring research engineering vs classical ML skills
• Assessing AI safety risks from autonomous R&D
• Comparing agent performance against human experts
• Research on AI-driven scientific discovery

Avoid When

• Standard ML benchmarking with public solutions
• Classical machine learning task evaluation
• Short-term capability assessment only
• Environments without GPU/compute resources
• Non-research engineering skill evaluation

Key Metrics

Expert Success Rate

82% of human experts achieved non-zero scores

Strong Solution Match

24% of experts matched or exceeded reference solutions

Short-term Agent Advantage

4x higher scores than humans at 2-hour budget

Long-term Human Advantage

2x human scores vs top agents at 32-hour budget

Solution Generation Speed

Agents generate/test solutions 10x faster than humans

Cost Efficiency

Much lower cost per solution attempt for AI agents

Top Use Cases

AI Safety Research: Evaluating R&D automation risks as highlighted in frontier AI safety policies

Research Capability Assessment: Measuring agent performance on scaling laws, GPU kernel optimization

Human-AI Comparison: Direct benchmarking against 71 expert attempts across multiple time budgets

Autonomous R&D Evaluation: Reproducing the published roster (Claude 3.5 Sonnet, o1-preview) on research tasks

Policy and Governance: Supporting White House NSM and EU AI Act evaluations for R&D capabilities

References & Further Reading

Deepen your understanding with these curated resources

Official METR RE-Bench Resources

RE-Bench: Evaluating Frontier AI R&D Capabilities (arXiv:2411.15114)

METR Blog: Evaluating R&D Capabilities of LLMs

METR AI R&D Evaluation Report (PDF)

RE-Bench ResearchGate Publication

Conference & Academic Recognition

ICML 2025 Poster: RE-Bench

OpenReview: RE-Bench Discussion

Hugging Face Paper Page

Hacker News Discussion: RE-Bench

Related METR Research

METR: Measuring Autonomous AI Capabilities

KernelBench: Can LLMs Write Efficient GPU Kernels?

METR Blog: Measuring Automated Kernel Engineering

Apollo Research: Forecasting Frontier Agent Capabilities

Policy & Safety Context

EU Artificial Intelligence Act

NVIDIA: GPU Kernel Generation with DeepSeek-R1

LessWrong: Forecasting Frontier Agent Capabilities

METR RE-Bench (RE-Bench) - Agentic Design