Loading patterns…
LLM as Judge(LJ)
Specific Producer-Critic implementation where an LLM acts as the critic to evaluate outputs
In 30 seconds
- What
- Generates multiple candidate outputs, then uses a second LLM to score each against a rubric and selects the highest-rated result.
- When to use
- You need consistent quality filtering and have budget for multiple LLM calls; ranking matters more than speed.
- Watch out
- The judge LLM may not reliably apply rubrics consistently, especially on subjective criteria or edge cases.
Ask the AI expert about this pattern
Opens the assistant with your question prefilled. You review it before sending.
LLM as Judge: Overview
Specific Producer-Critic implementation where an LLM acts as the critic to evaluate outputs
- Automated quality assessment
- Natural language evaluation
- Scalable ranking/scoring
- Configurable rubrics
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.
References
The papers, specifications, and repositories this pattern is based on.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)arXiv:2306.05685
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023)arXiv:2303.16634
- Large Language Models are not Fair Evaluators (Wang et al., 2023)arXiv:2305.17926
- Prometheus: Inducing Fine-grained Evaluation Capability in LLMs (Kim et al., 2024)arXiv:2310.08491
- OpenAI Evals Framework Documentation
From the engineer behind this catalog
Get your agent architecture reviewed
This page documents one pattern. Your system runs dozens, and most failures live in how they fit together. Have the whole design reviewed against the 288 patterns in this catalog: architecture, reliability, evaluation and cost, every finding mapped to the pattern that fixes it.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September