モデルアーキテクチャ
Multimodal fusion models
Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.
理解のためのモデル
“Multimodal” names the inputs and outputs, not one topology. Always ask where modalities are encoded, fused, and decoded.
データフロー
- Text / image / audio / video
- Modality encoders or tokenizers
- Projector, cross-attention, or shared backbone
- Joint representation / language decoder
- Text, media, or action output
学習方法
Systems combine contrastive alignment, captioning or next-token likelihood, masked objectives, paired instruction data, and sometimes separately pretrained frozen components.
推論の実行方法
Inputs are encoded into modality tokens or features, fused before or inside the language/generative backbone, and decoded into one or more modalities. Input understanding does not imply output generation.
強み
- Grounds language in visual or acoustic inputs
- Supports document, chart, image, video, and audio workflows
- Can reuse strong pretrained modality components
トレードオフ
- Modality imbalance and connector bottlenecks
- Media tokens consume substantial context and compute
- Evaluation must separate perception, grounding, reasoning, and generation
適する場合
- The task genuinely depends on non-text evidence
- Modality-specific slices and abstention are evaluated
- The architecture exposes enough detail for cost and privacy review
避ける・再検討する場合
- A transcript, OCR result, or structured extractor is sufficient
- “Supports images” is assumed to mean precise perception
- Sensitive media handling and retention are undefined
公開された方式の例
- • Flamingo cross-attention architecture
- • BLIP-2 learned querying connector
- • Shared-token multimodal decoder systems
よく組み合わせる要素
Vision / audio encodersDecoder-only language modelContrastive alignmentMedia generators