Agentic Design

Patterns
🪞

Self-RAG(SRAG)

Self-reflective RAG that adaptively determines retrieval necessity and evaluates retrieval quality through reflection tokens

Complexity: highKnowledge Retrieval (RAG)

🎯 30-Second Overview

Pattern: RAG system with adaptive retrieval decisions and self-reflection using trained reflection tokens for quality assessment

Why: Improves factual accuracy and reduces hallucinations through iterative self-critique and selective knowledge retrieval

Key Insight: Reflection tokens ([Retrieve], [IsRel], [IsSup], [IsUse]) enable models to assess retrieval necessity and response quality

⚡ Quick Implementation

1Retrieval Gate:Assess whether external knowledge is needed for the query
2Retrieve & Rank:Fetch relevant passages and apply neural reranking
3Generate Draft:Create initial response using retrieved context
4Self-Critique:Use reflection tokens to evaluate response quality
5Refine Output:Iteratively improve based on self-assessment
Example: query → retrieval_gate → retrieve → generate → self_critique → [refine] → final_response

📋 Do's & Don'ts

✅Train models with reflection tokens ([Retrieve], [IsRel], [IsSup], [IsUse])
✅Implement retrieval necessity prediction to avoid unnecessary context
✅Use calibrated confidence thresholds for triggering refinement
✅Enforce citation requirements with evidence grounding
✅Cache reflection outputs and retrieval results for efficiency
❌Allow unconstrained reflection tokens that become verbose
❌Skip validation of self-critique calibration against ground truth
❌Create infinite refinement loops without iteration limits
❌Rely solely on self-assessment without external validation
❌Ignore computational cost of multiple generation rounds

🚦 When to Use

Use When

  • • Factual accuracy and verifiability are critical requirements
  • • Domain expertise requires balancing parametric and retrieved knowledge
  • • Applications need confidence calibration and uncertainty quantification
  • • High-stakes decisions requiring explainable reasoning and citations
  • • Knowledge-intensive tasks in medical, legal, or scientific domains

Avoid When

  • • Simple queries where standard RAG provides sufficient accuracy
  • • Real-time applications with strict latency constraints
  • • Resource-constrained environments limiting multiple generation rounds
  • • Domains with insufficient training data for reliable self-critique
  • • Applications where citation overhead is unnecessary

📊 Key Metrics

Answer Faithfulness
Factual accuracy and consistency with retrieved evidence
Reflection Calibration
Correlation between confidence scores and actual accuracy
Retrieval Precision
Proportion of retrieved passages that are genuinely useful
Citation Coverage
Percentage of claims supported by retrieved evidence
Refinement Effectiveness
Quality improvement through iterative self-correction
Computational Efficiency
Quality gains per additional token or retrieval call

💡 Top Use Cases

Medical Q&A: Clinical decision support requiring high accuracy and evidence-based responses
Legal Research: Case law analysis with citation requirements and confidence assessment
Scientific Literature Review: Research synthesis with source attribution and uncertainty quantification
Financial Analysis: Investment research requiring balanced parametric and real-time market data
Educational Content: Academic tutoring with verified information and learning confidence tracking

References & Further Reading

Deepen your understanding with these curated resources

Foundational Papers & Self-RAG Research

Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (Asai et al., 2023)

SELF-REFINE: Iterative Refinement with Self-Feedback (Madaan et al., 2023)

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)

Teaching Language Models to Self-Correct via Reinforcement Learning (Welleck et al., 2023)

Reflection Tokens & Training Methods

Self-RAG Official Implementation: Reflection Token Training

Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022)

Training Language Models with Language Feedback at Scale (Scheurer et al., 2023)

Self-Consistency Improves Chain of Thought Reasoning (Wang et al., 2022)

Retrieval & Context Assessment

When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (Mallen et al., 2023)

Active Retrieval Augmented Generation (Jiang et al., 2023)

FiD: Fusion-in-Decoder for Open-Domain Question Answering (Izacard & Grave, 2021)

Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., 2020)

Implementation Frameworks & Tools

LangChain Self-Query Retrieval and Reflection Chains

LlamaIndex Self-Reflection Agent Implementation

Haystack Self-Correcting RAG Pipeline Tutorial

Transformers Library: Self-RAG Model Integration

Evaluation & Calibration Methods

RAGAS: Automated Evaluation of RAG Applications

TruLens: Evaluation and Observability for LLM Applications

Calibrating Sequence Likelihood Improves Conditional Language Generation (Zhao et al., 2022)

Teaching Models to Express Their Uncertainty in Words (Lin et al., 2022)

Production Deployment & Optimization

LangSmith: Monitoring and Evaluation for Self-RAG Systems

OpenAI Function Calling for Self-Critique Implementation

Anthropic Constitutional AI Training Methods

Weights & Biases: Experiment Tracking for Self-RAG

Contribute to this collection

Know a great resource? Submit a pull request to add it.

Contribute

Patterns

closed

Loading...

Built by Kortexya