Loading...
🔬
METR RE-Bench(RE-Bench)
Benchmark for measuring performance of frontier model agents on ML research engineering tasks, comparing against human expert capabilities.
Complexity: highEvaluation and Monitoring
🎯 30-Second Overview
Pattern: METR's benchmark comparing frontier AI agents against 71 human experts across 7 ML research engineering environments
Why: Evaluates R&D automation capabilities highlighted as key risk in frontier AI safety policies
Key Insight: Agents achieve 4x human performance at 2h budget but humans outperform 2x at 32h - time scaling matters
⚡ Quick Implementation
1Setup:Deploy 7 ML research engineering environments with GPU access
2Configure:Set time budgets (2h, 8h, 32h) and scoring functions
3Evaluate:Run agents on scaling laws, GPU kernels, and ML optimization tasks
4Compare:Benchmark against 71 human expert attempts (61 distinct experts)
5Analyze:Assess R&D automation capabilities vs human performance
Example: re_bench = REBench(environments=7, experts=71, models=[claude_3.5, o1_preview], budget="8h")
📋 Do's & Don'ts
✅Test across all 7 environments for comprehensive R&D capability assessment
✅Compare performance at multiple time budgets (2h, 8h, 32h)
✅Focus on open-ended research engineering tasks, not classical ML
✅Provide GPU access for kernel optimization and scaling experiments
✅Use human expert baselines from 61 distinct researchers
❌Expect agents to outperform humans on extended time budgets (32h)
❌Focus only on short-term performance - humans show better scaling
❌Use publicly available solutions - benchmark designed to avoid this
❌Skip safety considerations for R&D automation capabilities
❌Ignore the 10x speed advantage agents have in solution generation
🚦 When to Use
Use When
- • Evaluating frontier AI R&D automation capabilities
- • Measuring research engineering vs classical ML skills
- • Assessing AI safety risks from autonomous R&D
- • Comparing agent performance against human experts
- • Research on AI-driven scientific discovery
Avoid When
- • Standard ML benchmarking with public solutions
- • Classical machine learning task evaluation
- • Short-term capability assessment only
- • Environments without GPU/compute resources
- • Non-research engineering skill evaluation
📊 Key Metrics
Expert Success Rate
82% of human experts achieved non-zero scores
Strong Solution Match
24% of experts matched or exceeded reference solutions
Short-term Agent Advantage
4x higher scores than humans at 2-hour budget
Long-term Human Advantage
2x human scores vs top agents at 32-hour budget
Solution Generation Speed
Agents generate/test solutions 10x faster than humans
Cost Efficiency
Much lower cost per solution attempt for AI agents
💡 Top Use Cases
AI Safety Research: Evaluating R&D automation risks as highlighted in frontier AI safety policies
Research Capability Assessment: Measuring agent performance on scaling laws, GPU kernel optimization
Human-AI Comparison: Direct benchmarking against 71 expert attempts across multiple time budgets
Autonomous R&D Evaluation: Testing frontier models (Claude 3.5 Sonnet, o1-preview) on research tasks
Policy and Governance: Supporting White House NSM and EU AI Act evaluations for R&D capabilities
References & Further Reading
Deepen your understanding with these curated resources
Conference & Academic Recognition
Contribute to this collection
Know a great resource? Submit a pull request to add it.
🔬
METR RE-Bench(RE-Bench)
Benchmark for measuring performance of frontier model agents on ML research engineering tasks, comparing against human expert capabilities.
Complexity: highEvaluation and Monitoring
🎯 30-Second Overview
Pattern: METR's benchmark comparing frontier AI agents against 71 human experts across 7 ML research engineering environments
Why: Evaluates R&D automation capabilities highlighted as key risk in frontier AI safety policies
Key Insight: Agents achieve 4x human performance at 2h budget but humans outperform 2x at 32h - time scaling matters
⚡ Quick Implementation
1Setup:Deploy 7 ML research engineering environments with GPU access
2Configure:Set time budgets (2h, 8h, 32h) and scoring functions
3Evaluate:Run agents on scaling laws, GPU kernels, and ML optimization tasks
4Compare:Benchmark against 71 human expert attempts (61 distinct experts)
5Analyze:Assess R&D automation capabilities vs human performance
Example: re_bench = REBench(environments=7, experts=71, models=[claude_3.5, o1_preview], budget="8h")
📋 Do's & Don'ts
✅Test across all 7 environments for comprehensive R&D capability assessment
✅Compare performance at multiple time budgets (2h, 8h, 32h)
✅Focus on open-ended research engineering tasks, not classical ML
✅Provide GPU access for kernel optimization and scaling experiments
✅Use human expert baselines from 61 distinct researchers
❌Expect agents to outperform humans on extended time budgets (32h)
❌Focus only on short-term performance - humans show better scaling
❌Use publicly available solutions - benchmark designed to avoid this
❌Skip safety considerations for R&D automation capabilities
❌Ignore the 10x speed advantage agents have in solution generation
🚦 When to Use
Use When
- • Evaluating frontier AI R&D automation capabilities
- • Measuring research engineering vs classical ML skills
- • Assessing AI safety risks from autonomous R&D
- • Comparing agent performance against human experts
- • Research on AI-driven scientific discovery
Avoid When
- • Standard ML benchmarking with public solutions
- • Classical machine learning task evaluation
- • Short-term capability assessment only
- • Environments without GPU/compute resources
- • Non-research engineering skill evaluation
📊 Key Metrics
Expert Success Rate
82% of human experts achieved non-zero scores
Strong Solution Match
24% of experts matched or exceeded reference solutions
Short-term Agent Advantage
4x higher scores than humans at 2-hour budget
Long-term Human Advantage
2x human scores vs top agents at 32-hour budget
Solution Generation Speed
Agents generate/test solutions 10x faster than humans
Cost Efficiency
Much lower cost per solution attempt for AI agents
💡 Top Use Cases
AI Safety Research: Evaluating R&D automation risks as highlighted in frontier AI safety policies
Research Capability Assessment: Measuring agent performance on scaling laws, GPU kernel optimization
Human-AI Comparison: Direct benchmarking against 71 expert attempts across multiple time budgets
Autonomous R&D Evaluation: Testing frontier models (Claude 3.5 Sonnet, o1-preview) on research tasks
Policy and Governance: Supporting White House NSM and EU AI Act evaluations for R&D capabilities
References & Further Reading
Deepen your understanding with these curated resources
Conference & Academic Recognition
Contribute to this collection
Know a great resource? Submit a pull request to add it.