Архитектуры моделей
State-space & recurrent models
Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.
Ментальная модель
Stream the sequence through a learned state transition; the state is a compressed memory of the past.
Поток данных
- Current token + prior state
- Learned state update
- Selective gate or recurrent mixer
- Output hidden state
- Carry state forward
Как проходит обучение
Sequence modeling losses are paired with recurrence or structured state-space layers. Parallel scans or related formulations can make training more parallel than a naive step-by-step RNN.
Как выполняется инференс
Decode maintains a bounded recurrent state per layer. Hybrid architectures may alternate state-space, convolution, and attention layers to recover capabilities each mechanism handles well.
Сильные стороны
- Linear-time sequence processing in the core recurrence
- Compact streaming state at inference
- Natural fit for long or continuously arriving sequences
Компромиссы
- Compressed state can lose precise distant details
- Tooling and optimized kernels may be less mature than standard attention
- Architecture-level complexity claims must be verified end to end
Использовать, когда
- Streaming or long-sequence efficiency is a first-order requirement
- The target runtime has optimized kernels
- Recall, quality, and throughput are benchmarked on the real workload
Избегать или пересмотреть, когда
- Exact arbitrary retrieval from long context is assumed without testing
- The serving stack cannot exploit the architecture
- A mature Transformer deployment already meets the budget
Примеры опубликованных семейств
- • Mamba selective state-space models
- • RWKV-style recurrent language models
- • Hybrid attention–SSM stacks
Часто сочетается с
Decoder language modelsMixture of ExpertsHybrid attention layers