What happened: How to Loop MoE: Flatten the Experts, Untie the Attention

Researchers propose Foil, a method for looping mixture-of-experts models by flattening expert layers and giving each pass independent attention parameters.

Why it matters: How to Loop MoE: Flatten the Experts, Untie the Attention

Relevant for engineers optimizing sparse transformer architectures who want better expert utilization and parameter efficiency in large language models.

Watch out: How to Loop MoE: Flatten the Experts, Untie the Attention

Results shown at 20B and 100B tokens; unclear how findings scale to production-size training runs or whether gains persist across diverse downstream tasks.

In the news

How to Loop MoE: Flatten the Experts, Untie the Attention

arXiv cs.AI · Published September 28, 2026 · 1 min read

In 30 seconds

What happened: Researchers propose Foil, a method for looping mixture-of-experts models by flattening expert layers and giving each pass independent attention parameters.
Why it matters: Relevant for engineers optimizing sparse transformer architectures who want better expert utilization and parameter efficiency in large language models.
Watch out: Results shown at 20B and 100B tokens; unclear how findings scale to production-size training runs or whether gains persist across diverse downstream tasks.

Read the full article

attention
token
mixture-of-experts
phi

The patterns behind this

Each one covers how the technique works, when it earns its cost, and where it breaks.

The Agent Architect

One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.

Weekly email, one-click unsubscribe. We only use your address to send the briefing.