モデルアーキテクチャ
State-space & recurrent models
Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.
理解のためのモデル
Stream the sequence through a learned state transition; the state is a compressed memory of the past.
データフロー
- Current token + prior state
- Learned state update
- Selective gate or recurrent mixer
- Output hidden state
- Carry state forward
学習方法
Sequence modeling losses are paired with recurrence or structured state-space layers. Parallel scans or related formulations can make training more parallel than a naive step-by-step RNN.
推論の実行方法
Decode maintains a bounded recurrent state per layer. Hybrid architectures may alternate state-space, convolution, and attention layers to recover capabilities each mechanism handles well.
強み
- Linear-time sequence processing in the core recurrence
- Compact streaming state at inference
- Natural fit for long or continuously arriving sequences
トレードオフ
- Compressed state can lose precise distant details
- Tooling and optimized kernels may be less mature than standard attention
- Architecture-level complexity claims must be verified end to end
適する場合
- Streaming or long-sequence efficiency is a first-order requirement
- The target runtime has optimized kernels
- Recall, quality, and throughput are benchmarked on the real workload
避ける・再検討する場合
- Exact arbitrary retrieval from long context is assumed without testing
- The serving stack cannot exploit the architecture
- A mature Transformer deployment already meets the budget
公開された方式の例
- • Mamba selective state-space models
- • RWKV-style recurrent language models
- • Hybrid attention–SSM stacks
よく組み合わせる要素
Decoder language modelsMixture of ExpertsHybrid attention layers