Agentic Design

Patterns
๐Ÿง 

Meta-Learning Systems(MLS)

Learning how to learn efficiently across different tasks and domains through meta-optimization

Complexity: highLearning and Adaptation

๐ŸŽฏ 30-Second Overview

Pattern: Learn to learn: acquire meta-knowledge enabling rapid adaptation to new tasks with minimal data

Why: Enables few-shot learning, rapid domain adaptation, and efficient knowledge transfer across related tasks

Key Insight: Models learn optimization procedures and inductive biases that generalize across task distributions

โšก Quick Implementation

1Tasks:Define distribution of related learning tasks
2Meta-Train:Learn to adapt quickly across task distribution
3Support Set:Provide few examples of new target task
4Adapt:Rapidly specialize to new task using meta-knowledge
5Query:Perform target task with minimal examples
Example: task_distribution โ†’ meta_model โ†’ support_examples โ†’ adapted_model โ†’ predictions

๐Ÿ“‹ Do's & Don'ts

โœ…Design diverse meta-training task distributions
โœ…Balance task complexity and similarity in meta-training
โœ…Use proper train/validation/test task splits
โœ…Monitor for overfitting to meta-training tasks
โœ…Implement gradient-based and gradient-free approaches
โœ…Track adaptation speed and final performance metrics
โŒUse identical tasks in meta-training and meta-testing
โŒIgnore computational overhead of meta-optimization
โŒApply to domains with insufficient task diversity
โŒSkip ablation studies on meta-learning components
โŒUse inadequate support set sizes for evaluation

๐Ÿšฆ When to Use

Use When

  • โ€ข Many related tasks with limited data each
  • โ€ข Need rapid adaptation to new domains
  • โ€ข Tasks share underlying structure or patterns
  • โ€ข Few-shot learning requirements are critical
  • โ€ข Domain has natural task distribution

Avoid When

  • โ€ข Single task with abundant training data
  • โ€ข Tasks are completely unrelated
  • โ€ข Real-time inference constraints are severe
  • โ€ข Limited computational resources for meta-training
  • โ€ข Task distribution is poorly defined

๐Ÿ“Š Key Metrics

Few-Shot Accuracy
Performance with K examples per class
Adaptation Speed
Gradient steps to convergence
Transfer Efficiency
Performance vs baseline on new tasks
Meta-Generalization
Performance on unseen task types
Sample Efficiency
Examples needed for target performance
Computational Cost
FLOPs for meta-training and adaptation

๐Ÿ’ก Top Use Cases

Few-Shot Classification: Image recognition with limited labeled examples per class
Domain Adaptation: Rapidly adapt models to new domains with minimal data
Neural Architecture Search: Learn to design architectures for new tasks
Hyperparameter Optimization: Learn optimal hyperparameters across task families
Reinforcement Learning: Quick adaptation to new environments and reward structures
Natural Language Processing: Few-shot text classification and named entity recognition

References & Further Reading

Deepen your understanding with these curated resources

Foundational Papers

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (Finn et al., 2017)

Learning to Learn by Gradient Descent by Gradient Descent (Andrychowicz et al., 2016)

Matching Networks for One Shot Learning (Vinyals et al., 2016)

Prototypical Networks for Few-shot Learning (Snell et al., 2017)

MAML Variants & Extensions

On First-Order Meta-Learning Algorithms (Nichol et al., 2018)

How to train your MAML (Antoniou et al., 2018)

MAML++: A Domain-Agnostic Meta-Learning Algorithm (Antoniou et al., 2019)

Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace (Li et al., 2018)

Optimization-Based Meta-Learning

Meta-SGD: Learning to Learn Quickly for Few-Shot Learning (Li et al., 2017)

Learning to learn without gradient descent by gradient descent (Chen et al., 2016)

Optimization as a Model for Few-Shot Learning (Ravi & Larochelle, 2016)

Meta-Learning with Differentiable Convex Optimization (Lee et al., 2019)

Memory-Based Meta-Learning

Meta-Learning with Memory-Augmented Neural Networks (Santoro et al., 2016)

Neural Turing Machines (Graves et al., 2014)

Differentiable Neural Computers (Graves et al., 2016)

Learning to Remember Rare Events (Kaiser et al., 2017)

Meta-Learning Theory

A Theoretical Analysis of the Number of Shots in Few-Shot Learning (Cao et al., 2019)

Understanding and Improving Meta-Learning (Raghu et al., 2019)

Reconciling meta-learning and continual learning with online mixtures of tasks (Jerfel et al., 2019)

Meta-Learning: A Survey (Hospedales et al., 2020)

Recent Advances (2022-2024)

MetaICL: Learning to Learn In Context (Min et al., 2022)

In-Context Learning and Gradient-Based Meta-Learning (Akyรผrek et al., 2022)

Language Models are Few-Shot Learners (Brown et al., 2020)

Meta-Learning via Language Model In-context Tuning (Chen et al., 2022)

Neural Architecture Search & AutoML

DARTS: Differentiable Architecture Search (Liu et al., 2018)

Neural Architecture Search with Reinforcement Learning (Zoph & Le, 2016)

Auto-Meta: Automated Gradient Based Meta Learner Search (Kim et al., 2018)

MetaNAS: Meta Neural Architecture Search (Elsken et al., 2020)

Applications & Domains

Learning to Reinforcement Learn (Wang et al., 2016)

Meta-Learning for Few-Shot Natural Language Processing (Bansal et al., 2019)

Model-Agnostic Meta-Learning for Multilingual Machine Translation (Gu et al., 2018)

Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector (Fan et al., 2019)

Implementation Resources

learn2learn: A PyTorch Meta-Learning Library

MAML Implementation in TensorFlow

Torchmeta: A Meta-Learning library for PyTorch

Higher: A pytorch library for higher-order optimization

Datasets & Benchmarks

Omniglot Dataset for Few-Shot Learning

miniImageNet: A Dataset for Few-Shot Learning

Meta-Dataset: A Dataset of Datasets for Few-Shot Learning

VTAB: Visual Task Adaptation Benchmark

Tools & Frameworks

JAX-based MAML Implementation

TensorFlow Meta-Learning (TF-Agents)

PyTorch Lightning Meta-Learning

Avalanche: Continual Learning Library

Community & Research Groups

Meta-Learning Workshop (NeurIPS)

Few-Shot Learning Research Community

Berkeley AI Research (BAIR) - Meta-Learning

OpenAI Meta-Learning Research

Contribute to this collection

Know a great resource? Submit a pull request to add it.

Contribute

Patterns

closed

Loading...

Built by Kortexya