Agentic Design

Patterns
🧭

LLM-based Routing(LBR)

An intelligent query distribution system that uses a specialized LLM router to analyze incoming requests and dynamically route them to the most appropriate model, API endpoint, or processing pipeline based on query characteristics, ensuring optimal resource utilization and response quality through intent classification and capability matching

Complexity: mediumRouting

🎯 30-Second Overview

Pattern: Use LLM to analyze input and determine routing path dynamically

Why: Handles complex, nuanced routing decisions that simple rules can't capture

Key Insight: Prompt engineering + structured outputs = reliable intent classification

⚡ Quick Implementation

1Analyze:Prompt LLM to classify intent/category
2Extract:Parse structured output (JSON/enum)
3Map:Route decision → handler/agent
4Execute:Invoke selected workflow
5Monitor:Track routing accuracy & latency
Example: analyze_query → "category: booking" → booking_agent.invoke(query)

📋 Do's & Don'ts

✅Use structured outputs (JSON mode) for deterministic parsing
✅Provide clear examples in routing prompt
✅Implement fallback routes for unclear classifications
✅Cache routing decisions for identical queries
✅Use temperature=0 for consistent routing
❌Rely on free-form text parsing for routing
❌Skip validation of LLM routing output
❌Use high temperature for routing decisions
❌Ignore edge cases and ambiguous inputs
❌Route without confidence thresholds

🚦 When to Use

Use When

  • • Complex intent classification needed
  • • Natural language understanding required
  • • Dynamic routing rules that evolve
  • • Multi-dimensional routing criteria

Avoid When

  • • Simple keyword-based routing suffices
  • • Ultra-low latency requirements (<100ms)
  • • Deterministic routing is mandatory
  • • Cost constraints are tight

📊 Key Metrics

Routing Accuracy
% correctly routed queries
Classification Time
P50/P95 routing latency
Ambiguity Rate
% queries needing clarification
Cost per Route
LLM tokens × price
Fallback Rate
% routed to default handler
Cache Hit Rate
% reused routing decisions

💡 Top Use Cases

Customer Support: analyze intent → route to sales/tech/billing specialist
Multi-Tool Agents: parse request → select appropriate tool/API
Document Processing: classify type → apply correct parser/workflow
Query Routing: understand complexity → route to fast/powerful model
Workflow Selection: analyze task → choose sequential/parallel execution

References & Further Reading

Deepen your understanding with these curated resources

Academic Papers

RouteLLM: Learning to Route LLMs with Preference Data (2024)

Building Effective Agents - Anthropic (2024)

A Survey on LLM-based Multi-Agent Systems (2024)

Survey of Prompt Engineering Methods (2024)

Implementation Guides

LangChain Routing Documentation

OpenAI Structured Outputs for Routing

Google ADK Agent Routing

LangGraph Conditional Edges

Tools & Libraries

RouteLLM Framework - Cost-Effective LLM Routing

LangChain RunnableBranch

LlamaIndex Router Query Engine

Semantic Router Library

Community & Discussions

LMSYS RouteLLM Blog Post

IBM Research - LLM Routing for Quality

LangChain Discord - Routing Discussions

Reddit r/LocalLLaMA - Routing Strategies

Contribute to this collection

Know a great resource? Submit a pull request to add it.

Contribute

Patterns

closed

Loading...