Model Architectures
State-space & recurrent models
Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.
Mental model
Stream the sequence through a learned state transition; the state is a compressed memory of the past.
Data flow
- Current token + prior state
- Learned state update
- Selective gate or recurrent mixer
- Output hidden state
- Carry state forward
How it trains
Sequence modeling losses are paired with recurrence or structured state-space layers. Parallel scans or related formulations can make training more parallel than a naive step-by-step RNN.
How inference runs
Decode maintains a bounded recurrent state per layer. Hybrid architectures may alternate state-space, convolution, and attention layers to recover capabilities each mechanism handles well.
Strengths
- Linear-time sequence processing in the core recurrence
- Compact streaming state at inference
- Natural fit for long or continuously arriving sequences
Trade-offs
- Compressed state can lose precise distant details
- Tooling and optimized kernels may be less mature than standard attention
- Architecture-level complexity claims must be verified end to end
Use it when
- Streaming or long-sequence efficiency is a first-order requirement
- The target runtime has optimized kernels
- Recall, quality, and throughput are benchmarked on the real workload
Avoid or challenge it when
- Exact arbitrary retrieval from long context is assumed without testing
- The serving stack cannot exploit the architecture
- A mature Transformer deployment already meets the budget
Illustrative published families
- • Mamba selective state-space models
- • RWKV-style recurrent language models
- • Hybrid attention–SSM stacks
Commonly combines with
Decoder language modelsMixture of ExpertsHybrid attention layers