Model Architectures
Multimodal fusion models
Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.
Mental model
“Multimodal” names the inputs and outputs, not one topology. Always ask where modalities are encoded, fused, and decoded.
Data flow
- Text / image / audio / video
- Modality encoders or tokenizers
- Projector, cross-attention, or shared backbone
- Joint representation / language decoder
- Text, media, or action output
How it trains
Systems combine contrastive alignment, captioning or next-token likelihood, masked objectives, paired instruction data, and sometimes separately pretrained frozen components.
How inference runs
Inputs are encoded into modality tokens or features, fused before or inside the language/generative backbone, and decoded into one or more modalities. Input understanding does not imply output generation.
Strengths
- Grounds language in visual or acoustic inputs
- Supports document, chart, image, video, and audio workflows
- Can reuse strong pretrained modality components
Trade-offs
- Modality imbalance and connector bottlenecks
- Media tokens consume substantial context and compute
- Evaluation must separate perception, grounding, reasoning, and generation
Use it when
- The task genuinely depends on non-text evidence
- Modality-specific slices and abstention are evaluated
- The architecture exposes enough detail for cost and privacy review
Avoid or challenge it when
- A transcript, OCR result, or structured extractor is sufficient
- “Supports images” is assumed to mean precise perception
- Sensitive media handling and retention are undefined
Illustrative published families
- • Flamingo cross-attention architecture
- • BLIP-2 learned querying connector
- • Shared-token multimodal decoder systems
Commonly combines with
Vision / audio encodersDecoder-only language modelContrastive alignmentMedia generators