Agentic Design

Patterns
🧠

LLM Checkpoint Recovery (Mnemosyne)(LCR)

Lightweight device proxy architecture for LLM fault recovery with just-in-time checkpointing and partial topology reconstruction

Complexity: highFault Tolerance Infrastructure

🎯 30-Second Overview

Pattern: Device proxy architecture with just-in-time checkpointing for LLM fault recovery

Why: Reduces recovery overhead by 58.8% vs traditional approaches, enables partial topology reconstruction

Key Insight: Lightweight device proxies + flexible CCL + incremental communication reinitialization = 3.6% daily overhead

⚡ Quick Implementation

1Setup Proxy:Deploy device proxy layer for error interception
2Configure CCL:Initialize flexible collective communication library
3Enable JIT:Activate just-in-time checkpointing triggers
4Partial Recovery:Implement incremental topology reconstruction
5Monitor:Track failure patterns and recovery times
Example: device_proxy → failure_detection → checkpoint_trigger → partial_reconstruction → resume_training

📋 Do's & Don'ts

✅Use lightweight device proxies optimized for fault tolerance
✅Implement partial topology reconstruction around failed nodes
✅Cache gradient states and optimizer checkpoints separately
✅Use incremental communication reinitialization (not full restart)
✅Monitor memory usage patterns to predict failures
❌Rely on elastic training features for pure fault tolerance
❌Perform global communication reinitialization on single failures
❌Store massive checkpoints to slow storage during training
❌Ignore temporal dominance of communication overhead
❌Use generic checkpointing for 70B+ parameter models

🚦 When to Use

Use When

  • • Distributed LLM training (7B+ parameters)
  • • Multi-week training cycles
  • • High failure rate environments
  • • Limited checkpoint storage bandwidth

Avoid When

  • • Small model training (<1B parameters)
  • • Single-node deployments
  • • Stable hardware environments
  • • Short training cycles (<24h)

📊 Key Metrics

Recovery Time
Seconds to resume training
Overhead
% daily training time lost
Checkpoint Size
GB per model snapshot
Failure Detection
Time to detect node failure
Memory Utilization
% peak memory preserved
Communication Cost
Bandwidth for reconstruction

💡 Top Use Cases

LLM Pre-training: 70B+ parameter models with week-long training cycles
Distributed Fine-tuning: Multi-node adaptation with frequent hardware failures
MoE Training: Sparse mixture-of-experts with massive checkpoint sizes
Hybrid Parallelism: Data/tensor/pipeline parallel combinations with complex failure modes
Cloud Training: Spot instance training with predictable interruptions

References & Further Reading

Deepen your understanding with these curated resources

Core Academic Research (2024)

Mnemosyne: Lightweight and Fast Error Recovery for LLM Training (Asia-Pacific Workshop on Networking 2024)

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey (July 2024)

Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing (August 2024)

MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training (2024)

Technical Foundations

Gemini: Fast Failure Recovery in Distributed Training (SOSP 2023)

REFT: Reliable and Efficient Fault Tolerance for Hybrid Parallel Training

Amazon Science: More-efficient recovery from failures during large-ML-model training

VAST Data: A Checkpoint on Checkpoints in Large Language Models

Implementation Frameworks

DeepSpeed Checkpointing and ZeRO Optimizer States

PyTorch FSDP: Fully Sharded Data Parallel with Checkpointing

Megatron-LM: Checkpointing for Large Transformer Models

FairScale: Facebook AI Research Scaling Library

Industry & Community

Gradient AI: Distributed LLM Training - Orchestration & Fault Tolerance

MLOps for Large Models: Checkpointing & Fault Tolerance Course

ML Systems Papers: Curated Collection on Fault Tolerance

Demystifying Distributed Checkpointing (Technical Blog)

Contribute to this collection

Know a great resource? Submit a pull request to add it.

Contribute

Patterns

closed

Loading...

Built by Kortexya