MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

In 30 Sekunden

Was passiert ist

MuonGS, a new optimizer for LLM pretraining, uses Gram-Schmidt projection to control weight growth while allowing more flexible norm constraints than prior Muon variants.

Warum es zählt

Matters for engineers optimizing large language model training who want faster convergence and better long-horizon performance with existing hardware budgets.

Achtung

Paper shows fifteen percent token-equivalent speedup on tested models, but real-world gains depend on implementation details and may not generalize to all architectures or scales.