Pattern Evaluation Lab
Run a small test set through different agent patterns and different models, side by side. Every run reports real output, latency, token counts, and cost; an AI judge scores quality against your expected answers, and the whole comparison exports as JSON or CSV. Pattern choices rest on measurements rather than intuition.
One real comparison
The same task, four ways
Two models and two patterns on one question, measured rather than assumed. Latency is wall clock, token counts come from the provider, and cost is what was actually billed.
Task: A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
| Model | Pattern | Answer | Latency | Tokens | Cost |
|---|---|---|---|---|---|
| anthropic/claude-sonnet-5 | Direct | $0.05 | 3,065 ms | 63 | $0.000174 |
| anthropic/claude-sonnet-5 | Chain of Thought | $0.05 | 4,870 ms | 366 | $0.0031 |
| google/gemini-2.5-flash | Direct | $0.05 | 458 ms | 45 | $0.000024 |
| google/gemini-2.5-flash | Chain of Thought | $0.05 | 1,257 ms | 276 | $0.00058 |
All four runs answered correctly, so on this task the reasoning pattern bought nothing except time and money: the dearest run cost about 128 times the cheapest. That is the case for measuring a pattern choice instead of assuming it, and it is why the lab reports cost and latency next to the output.
Measured on 2026-08-06
And a judge that shows its work
The lab also scores quality with an AI judge, grounded in the expected answer you give it. In this captured comparison both runs computed the right number, and the judge still separated them: the task asked for the number first, and the run that buried it lost instruction-adherence points.
Task: A retail API rate limiter allows 120 requests per minute per key. A batch job needs to send 4,500 requests. What is the minimum time in minutes to send them all, and how should the job pace itself? Answer with the number first.
Expected answer: 37.5 minutes (4500 / 120), pacing at or under 2 requests per second.
| Model | Pattern | Quality | Adherence | Latency | Cost |
|---|---|---|---|---|---|
| openai/gpt-5-mini | Chain of Thought | 5.00/5 | 5/5 | 14,220 ms | $0.0021 |
| anthropic/claude-haiku-4-5 | Chain of Thought | 4.67/5 | 3/5 | 4,688 ms | $0.0023 |
The judge on the docked run: “The calculation and pacing advice are correct and clearly explained, but the answer is not given first as instructed; it only appears at the end after several intermediate steps.”
Head-to-head verdicts are checked for position bias: the judge is asked twice with the order swapped, and on this close pair it changed its pick, so the lab reported a tie instead of inventing a winner.
Scored 1 to 5 by anthropic/claude-sonnet-5 against the expected answer. Measured on 2026-08-10