Architectures de modèles
Encoder–decoder Transformers
Encode an input bidirectionally, then generate an output causally while cross-attending to the encoded source.
Modèle mental
A reader and a writer connected by cross-attention.
Flux de données
- Source sequence
- Bidirectional encoder
- Source memory
- Causal decoder + cross-attention
- Target sequence
Entraînement
Teacher-forced target-token likelihood is common. Text-to-text denoising or span corruption lets many tasks share the same input/output interface.
Exécution de l’inférence
The source is encoded once; the decoder generates the target autoregressively while attending to stable source states. Beam search is useful in some bounded tasks but is not universally best.
Atouts
- Clear separation between source understanding and target generation
- Strong fit for translation, summarization, transcription, and transformation
- Source memory is reused throughout decoding
Compromis
- Two stacks can increase parameters and deployment complexity
- Target decoding remains serial
- May be less convenient than one decoder-only interface for heterogeneous chat tasks
À utiliser lorsque
- Output is tightly conditioned on a distinct input
- Source and target have different structures or modalities
- Faithful transformation matters more than open-ended continuation
À éviter ou remettre en question lorsque
- A single general chat interface is the overriding requirement
- No distinct source sequence exists
- A non-generative encoder head is sufficient
Familles publiées à titre d’exemple
- • Original Transformer
- • T5
- • Whisper speech recognition architecture
Souvent combinée avec
Text transformationAudio / speechMultimodal fusion