Architectures de modèles
Multimodal fusion models
Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.
Modèle mental
“Multimodal” names the inputs and outputs, not one topology. Always ask where modalities are encoded, fused, and decoded.
Flux de données
- Text / image / audio / video
- Modality encoders or tokenizers
- Projector, cross-attention, or shared backbone
- Joint representation / language decoder
- Text, media, or action output
Entraînement
Systems combine contrastive alignment, captioning or next-token likelihood, masked objectives, paired instruction data, and sometimes separately pretrained frozen components.
Exécution de l’inférence
Inputs are encoded into modality tokens or features, fused before or inside the language/generative backbone, and decoded into one or more modalities. Input understanding does not imply output generation.
Atouts
- Grounds language in visual or acoustic inputs
- Supports document, chart, image, video, and audio workflows
- Can reuse strong pretrained modality components
Compromis
- Modality imbalance and connector bottlenecks
- Media tokens consume substantial context and compute
- Evaluation must separate perception, grounding, reasoning, and generation
À utiliser lorsque
- The task genuinely depends on non-text evidence
- Modality-specific slices and abstention are evaluated
- The architecture exposes enough detail for cost and privacy review
À éviter ou remettre en question lorsque
- A transcript, OCR result, or structured extractor is sufficient
- “Supports images” is assumed to mean precise perception
- Sensitive media handling and retention are undefined
Familles publiées à titre d’exemple
- • Flamingo cross-attention architecture
- • BLIP-2 learned querying connector
- • Shared-token multimodal decoder systems
Souvent combinée avec
Vision / audio encodersDecoder-only language modelContrastive alignmentMedia generators