Архитектуры моделей
Mixture of Experts (MoE)
Replace selected dense sublayers with many expert networks and a learned router that activates only a small subset for each token.
Ментальная модель
Large parameter capacity with sparse per-token activation; sparse compute is not the same as small memory or simple serving.
Поток данных
- Token hidden state
- Router scores experts
- Top-k expert dispatch
- Expert transformations
- Weighted merge + residual path
Как проходит обучение
The main task loss trains experts and router together. Load-balancing or routing regularizers discourage collapse into a few experts; distributed implementations must coordinate token dispatch.
Как выполняется инференс
Each token visits selected experts. Arithmetic can remain sparse while model weights, inter-device communication, routing imbalance, and batching still shape latency and cost.
Сильные стороны
- More parameter capacity without activating every parameter per token
- Experts may specialize through learned routing
- Scales naturally inside Transformer feed-forward blocks
Компромиссы
- Large weight memory and distributed communication
- Routing imbalance and capacity overflow complicate training
- Reported total parameters do not reveal active compute or serving efficiency
Использовать, когда
- Large-scale training and serving infrastructure can exploit sparse routing
- Capacity is more constrained than per-token arithmetic
- You can benchmark the actual hardware topology
Избегать или пересмотреть, когда
- A single-device or memory-constrained deployment is required
- “Sparse” is being assumed to guarantee lower latency
- Operational simplicity is more valuable than extra capacity
Примеры опубликованных семейств
- • Switch Transformer
- • Sparse expert layers inside language or multimodal models
Часто сочетается с
Decoder-only TransformerEncoder-decoder TransformerDiffusion Transformer