Model Architectures
Encoder–decoder Transformers
Encode an input bidirectionally, then generate an output causally while cross-attending to the encoded source.
Mental model
A reader and a writer connected by cross-attention.
Data flow
- Source sequence
- Bidirectional encoder
- Source memory
- Causal decoder + cross-attention
- Target sequence
How it trains
Teacher-forced target-token likelihood is common. Text-to-text denoising or span corruption lets many tasks share the same input/output interface.
How inference runs
The source is encoded once; the decoder generates the target autoregressively while attending to stable source states. Beam search is useful in some bounded tasks but is not universally best.
Strengths
- Clear separation between source understanding and target generation
- Strong fit for translation, summarization, transcription, and transformation
- Source memory is reused throughout decoding
Trade-offs
- Two stacks can increase parameters and deployment complexity
- Target decoding remains serial
- May be less convenient than one decoder-only interface for heterogeneous chat tasks
Use it when
- Output is tightly conditioned on a distinct input
- Source and target have different structures or modalities
- Faithful transformation matters more than open-ended continuation
Avoid or challenge it when
- A single general chat interface is the overriding requirement
- No distinct source sequence exists
- A non-generative encoder head is sufficient
Illustrative published families
- • Original Transformer
- • T5
- • Whisper speech recognition architecture
Commonly combines with
Text transformationAudio / speechMultimodal fusion