Loading patterns…
LLM Checkpoint Recovery (Mnemosyne)(LCR)
Lightweight device proxy architecture for LLM fault recovery with just-in-time checkpointing and partial topology reconstruction
In 30 seconds
- What
- Maintains lightweight device proxies that capture gradient and optimizer state on-demand, then reconstruct the training topology and resume from the last checkpoint after failures.
- When to use
- Large-scale distributed LLM training where frequent failures occur and you need sub-minute recovery without full retraining or elastic scaling.
- Watch out
- Just-in-time checkpointing adds latency during normal training; misconfigured triggers can cause cascading failures or memory exhaustion under high fault rates.
Ask the AI expert about this pattern
Opens the assistant with your question prefilled. You review it before sending.
LLM Checkpoint Recovery (Mnemosyne): Overview
Lightweight device proxy architecture for LLM fault recovery with just-in-time checkpointing and partial topology reconstruction
- Device proxy architecture
- Just-in-time checkpointing
- Flexible collective communication library
- Runtime link adjustment
- Partial topology reconstruction
- Memory state preservation
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.
References
The papers, specifications, and repositories this pattern is based on.
- Mnemosyne: Lightweight and Fast Error Recovery for LLM Training (Asia-Pacific Workshop on Networking 2024)
- Efficient Training of Large Language Models on Distributed Infrastructures: A Survey (July 2024)
- Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing (August 2024)
- MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training (2024)
- Gemini: Fast Failure Recovery in Distributed Training (SOSP 2023)
From the engineer behind this catalog
Get your agent architecture reviewed
This page documents one pattern. Your system runs dozens, and most failures live in how they fit together. Have the whole design reviewed against the 288 patterns in this catalog: architecture, reliability, evaluation and cost, every finding mapped to the pattern that fixes it.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September