能够处理并整合文本、图像、音频、视频和结构化数据源的检索增强生成
Loading technique guide…
模式： 使用专用编码器跨文本、图像、音频与视频检索并融合证据的多模态 RAG 系统
原因： 通过整合纯文本系统会遗漏的多模态信息,实现全面理解
关键洞察： 按模态配置编码器(CLIP、Whisper、BLIP-2),并以学习式融合权重整合跨模态证据
通过这些精选资源加深理解
Whisper: Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2022)
Speech-T5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (Ao et al., 2022)
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (Chen et al., 2022)
ImageBind: One Embedding Space To Bind Them All (Girdhar et al., 2023)
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations (Lu et al., 2019)
LXMERT: Learning Cross-Modality Encoder Representations from Transformers (Tan & Bansal, 2019)
Flamingo: a Visual Language Model for Few-Shot Learning (Alayrac et al., 2022)
ALIGN: Scaling Up Visual and Vision-Language Representation Learning (Jia et al., 2021)