模型架构
Multimodal fusion models
Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.
思维模型
“Multimodal” names the inputs and outputs, not one topology. Always ask where modalities are encoded, fused, and decoded.
数据流
- Text / image / audio / video
- Modality encoders or tokenizers
- Projector, cross-attention, or shared backbone
- Joint representation / language decoder
- Text, media, or action output
训练方式
Systems combine contrastive alignment, captioning or next-token likelihood, masked objectives, paired instruction data, and sometimes separately pretrained frozen components.
推理运行方式
Inputs are encoded into modality tokens or features, fused before or inside the language/generative backbone, and decoded into one or more modalities. Input understanding does not imply output generation.
优势
- Grounds language in visual or acoustic inputs
- Supports document, chart, image, video, and audio workflows
- Can reuse strong pretrained modality components
权衡
- Modality imbalance and connector bottlenecks
- Media tokens consume substantial context and compute
- Evaluation must separate perception, grounding, reasoning, and generation
适用场景
- The task genuinely depends on non-text evidence
- Modality-specific slices and abstention are evaluated
- The architecture exposes enough detail for cost and privacy review
应避免或质疑的场景
- A transcript, OCR result, or structured extractor is sufficient
- “Supports images” is assumed to mean precise perception
- Sensitive media handling and retention are undefined
已发表的示例系列
- • Flamingo cross-attention architecture
- • BLIP-2 learned querying connector
- • Shared-token multimodal decoder systems
常见组合
Vision / audio encodersDecoder-only language modelContrastive alignmentMedia generators