Agentic Design

Patterns
🎯

In-Context Learning(ICL)

Learning new tasks from examples in the input context without parameter updates (includes few-shot and zero-shot learning)

Complexity: mediumLearning and Adaptation

🎯 30-Second Overview

Pattern: Provide task examples in prompt to enable model learning without parameter updates

Why: Enables rapid task adaptation, requires no training, leverages model's pattern recognition for immediate performance

Key Insight: Models can learn from demonstrations in context, performing implicit gradient descent during inference

⚡ Quick Implementation

1Select:Choose relevant examples for task context
2Format:Structure examples with input-output pairs
3Prompt:Combine examples with target query
4Infer:Model learns pattern from examples
5Generate:Apply learned pattern to new input
Example: examples + query → model_inference → output

📋 Do's & Don'ts

✅Use diverse, high-quality examples representative of task
✅Maintain consistent formatting across all examples
✅Order examples from simple to complex when possible
✅Include edge cases and boundary conditions
✅Test with different example counts (1-shot to few-shot)
✅Use clear separators between examples
❌Use contradictory or inconsistent examples
❌Overload context with too many examples
❌Use biased or non-representative examples
❌Mix different task types in same prompt
❌Ignore example selection and ordering effects

🚦 When to Use

Use When

  • • Quick adaptation to new tasks needed
  • • Limited or no training data available
  • • Task requires demonstration over description
  • • Rapid prototyping and experimentation
  • • Model needs to understand complex patterns

Avoid When

  • • Large amounts of training data available
  • • Task requires extensive domain knowledge
  • • Context window limitations are severe
  • • High precision requirements exceed ICL capability
  • • Consistent performance across variations needed

📊 Key Metrics

Few-Shot Accuracy
Performance with K examples
Example Efficiency
Performance gain per example
Context Utilization
Token usage vs performance
Task Transfer
Generalization to unseen inputs
Example Sensitivity
Performance variance across example sets
Ordering Robustness
Stability across example orders

💡 Top Use Cases

Text Classification: Sentiment analysis, topic categorization with labeled examples
Data Extraction: Named entity recognition, information extraction from documents
Format Translation: JSON to XML, structured data transformation
Question Answering: Domain-specific QA with example question-answer pairs
Code Generation: Programming tasks with input-output code examples
Creative Writing: Style transfer, content generation with stylistic examples

References & Further Reading

Deepen your understanding with these curated resources

Foundational Papers

Language Models are Few-Shot Learners (Brown et al., 2020)

Learning to Retrieve Prompts for In-Context Learning (Rubin et al., 2021)

What Makes Good In-Context Examples for GPT-3? (Liu et al., 2021)

Calibrate Before Use: Improving Few-Shot Performance (Zhao et al., 2021)

Theoretical Understanding (2022-2024)

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent (Dai et al., 2022)

Transformers Learn In-Context by Gradient Descent (von Oswald et al., 2022)

In-context Learning and Induction Heads (Olsson et al., 2022)

What learning algorithm is in-context learning? Investigations with linear models (Akyürek et al., 2022)

Example Selection & Optimization

Learning To Retrieve Prompts for In-Context Learning (Rubin et al., 2021)

In-Context Example Selection with Influences (Nguyen & Wong, 2023)

Active Example Selection for In-Context Learning (Zhang et al., 2022)

Compositional Exemplars for In-context Learning (Ye et al., 2023)

Chain-of-Thought & Reasoning

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022)

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (Zhou et al., 2022)

Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al., 2022)

Large Language Models are Zero-Shot Reasoners (Kojima et al., 2022)

Recent Advances (2023-2024)

In-Context Learning Creates Task Vectors (Hendel et al., 2023)

Many-Shot In-Context Learning (Agarwal et al., 2024)

In-Context Learning with Long Context (Bertsch et al., 2024)

Understanding In-Context Learning via Supportive Pretraining Data (Shin et al., 2024)

Multimodal In-Context Learning

Flamingo: a Visual Language Model for Few-Shot Learning (Alayrac et al., 2022)

Multimodal Few-Shot Learning with Frozen Language Models (Tsimpoukelli et al., 2021)

GPT-4V(ision) System Card (OpenAI, 2023)

Visual In-Context Learning for Large Vision-Language Models (Yang et al., 2023)

Implementation Resources

LangChain Few-Shot Prompt Templates

OpenAI Best Practices for Prompt Engineering

Anthropic Prompt Engineering Guide

HuggingFace Transformers In-Context Learning

Datasets & Benchmarks

LAMA: LAnguage Model Analysis (Petroni et al., 2019)

BIG-bench: Beyond the Imitation Game Benchmark

HELM: Holistic Evaluation of Language Models

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding

Tools & Libraries

Guidance: A guidance language for controlling large language models

DSPy: Programming—not prompting—foundation models

PromptSource: An Integrated Development Environment for Natural Language Prompts

OpenPrompt: An Open-Source Framework for Prompt-learning

Analysis & Evaluation

Measuring and Narrowing the Compositionality Gap in Language Models (Press et al., 2022)

Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (Min et al., 2022)

Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations (Yoo et al., 2022)

Fantastically Ordered Prompts and Where to Find Them (Lu et al., 2021)

Community & Discussion

r/MachineLearning - In-Context Learning Discussions

AI Alignment Forum - In-Context Learning

EleutherAI Discord - Prompt Engineering Channel

Prompt Engineering Discord Community

Contribute to this collection

Know a great resource? Submit a pull request to add it.

Contribute

Patterns

closed

Loading...

Built by Kortexya