What happened: MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

MuonGS, a new optimizer for LLM pretraining, uses Gram-Schmidt projection to control weight growth while allowing more flexible norm constraints than prior Muon variants.

Why it matters: MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

Matters for engineers optimizing large language model training who want faster convergence and better long-horizon performance with existing hardware budgets.

In the news

MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

arXiv cs.AI · Published October 27, 2026 · 3 min read

In 30 seconds

What happened: MuonGS, a new optimizer for LLM pretraining, uses Gram-Schmidt projection to control weight growth while allowing more flexible norm constraints than prior Muon variants.
Why it matters: Matters for engineers optimizing large language model training who want faster convergence and better long-horizon performance with existing hardware budgets.
Watch out: Paper shows fifteen percent token-equivalent speedup on tested models, but real-world gains depend on implementation details and may not generalize to all architectures or scales.

Read the full article

llm
language model

The patterns behind this

Each one covers how the technique works, when it earns its cost, and where it breaks.

The Agent Architect

One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.

Weekly email, one-click unsubscribe. We only use your address to send the briefing.