模型架构
State-space & recurrent models
Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.
思维模型
Stream the sequence through a learned state transition; the state is a compressed memory of the past.
数据流
- Current token + prior state
- Learned state update
- Selective gate or recurrent mixer
- Output hidden state
- Carry state forward
训练方式
Sequence modeling losses are paired with recurrence or structured state-space layers. Parallel scans or related formulations can make training more parallel than a naive step-by-step RNN.
推理运行方式
Decode maintains a bounded recurrent state per layer. Hybrid architectures may alternate state-space, convolution, and attention layers to recover capabilities each mechanism handles well.
优势
- Linear-time sequence processing in the core recurrence
- Compact streaming state at inference
- Natural fit for long or continuously arriving sequences
权衡
- Compressed state can lose precise distant details
- Tooling and optimized kernels may be less mature than standard attention
- Architecture-level complexity claims must be verified end to end
适用场景
- Streaming or long-sequence efficiency is a first-order requirement
- The target runtime has optimized kernels
- Recall, quality, and throughput are benchmarked on the real workload
应避免或质疑的场景
- Exact arbitrary retrieval from long context is assumed without testing
- The serving stack cannot exploit the architecture
- A mature Transformer deployment already meets the budget
已发表的示例系列
- • Mamba selective state-space models
- • RWKV-style recurrent language models
- • Hybrid attention–SSM stacks
常见组合
Decoder language modelsMixture of ExpertsHybrid attention layers