Modellarchitekturen
Multimodal fusion models
Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.
Denkmodell
“Multimodal” names the inputs and outputs, not one topology. Always ask where modalities are encoded, fused, and decoded.
Datenfluss
- Text / image / audio / video
- Modality encoders or tokenizers
- Projector, cross-attention, or shared backbone
- Joint representation / language decoder
- Text, media, or action output
So wird trainiert
Systems combine contrastive alignment, captioning or next-token likelihood, masked objectives, paired instruction data, and sometimes separately pretrained frozen components.
So läuft die Inferenz
Inputs are encoded into modality tokens or features, fused before or inside the language/generative backbone, and decoded into one or more modalities. Input understanding does not imply output generation.
Stärken
- Grounds language in visual or acoustic inputs
- Supports document, chart, image, video, and audio workflows
- Can reuse strong pretrained modality components
Zielkonflikte
- Modality imbalance and connector bottlenecks
- Media tokens consume substantial context and compute
- Evaluation must separate perception, grounding, reasoning, and generation
Geeignet, wenn
- The task genuinely depends on non-text evidence
- Modality-specific slices and abstention are evaluated
- The architecture exposes enough detail for cost and privacy review
Vermeiden oder hinterfragen, wenn
- A transcript, OCR result, or structured extractor is sufficient
- “Supports images” is assumed to mean precise perception
- Sensitive media handling and retention are undefined
Beispielhafte veröffentlichte Familien
- • Flamingo cross-attention architecture
- • BLIP-2 learned querying connector
- • Shared-token multimodal decoder systems
Häufig kombiniert mit
Vision / audio encodersDecoder-only language modelContrastive alignmentMedia generators