In the news
Announcing vllm-metal: Concurrent Serving on Apple Silicon
vLLM · Ranran Haoran Zhang, Lik Xun Yuan, Chao Ju Chen, Eric Curtin, Michael Goin · Published · 3 min read
In 30 seconds
- What happened
- vLLM released vllm-metal v0.28.0, bringing concurrent LLM serving to Apple Silicon Macs with paged KV cache and OpenAI-compatible API.
- Why it matters
- Engineers running multiple overlapping inference requests on M1-M5 Macs need predictable latency, memory control, and batching beyond single-request inference.
- Watch out
- Speculative decoding with MTP requires greedy sampling and synchronous scheduling; prefix caching for hybrid models remains experimental and incompatible with speculative decoding.
- agent
- llm
- serving
- vllm
The patterns behind this
Each one covers how the technique works, when it earns its cost, and where it breaks.
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.