In the news
HiCache Across Multi-Parallelism and Distributed Serving
LMSYS · Published · 3 min read
In 30 seconds
- What happened
- HiCache extends KV cache reuse to host memory and external storage while maintaining consistency across context parallelism, pipeline parallelism, speculative decoding, and prefill-decode disaggregation.
- Why it matters
- Engineers deploying large language models with long-context agentic workloads where repeated request prefixes create cache reuse opportunities across distributed serving infrastructure.
- Watch out
- Per-layer KV gathering adds overhead; CP sharding optimization is ongoing. Consistency across pipeline stages requires careful synchronization of asynchronous transfers to avoid tensor shape mismatches.
- serving
The patterns behind this
- Speculative & Parallel Tool Execution
- Distributed Memory Architectures
- Agentic Context Engineering (Evolving Playbook)
Each one covers how the technique works, when it earns its cost, and where it breaks.
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.