In the news
Humming in SGLang: Flexible Low-Bit Kernels for Efficient MoE Serving
LMSYS · Published · 3 min read
In 30 seconds
- What happened
- Humming 0.1.18 adds flexible low-bit quantized kernels for MoE inference in SGLang, supporting MXFP4 weights and FP8 activations on Hopper and Blackwell GPUs.
- Why it matters
- Engineers deploying large mixture-of-experts models need efficient serving with reduced memory and compute requirements while maintaining model quality.
- Watch out
- Scale-range adjustment affected 11.27% of draft groups due to padding issues; a fix exists upstream but requires a follow-up Humming dependency upgrade.
- kernel
- serving
The patterns behind this
Each one covers how the technique works, when it earns its cost, and where it breaks.
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.