模型架构
Vision encoders
Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.
思维模型
A visual reader. It creates representations; generation requires a decoder or generative model.
数据流
- Image or frames
- Patch, convolutional, or hybrid stem
- Vision backbone
- Spatial tokens / feature pyramid / pooled vector
- Task head or modality connector
训练方式
Supervised labels, masked-image modeling, self-distillation, reconstruction, and image–text contrastive objectives produce different visual invariances and levels of spatial detail.
推理运行方式
The encoder runs once per image or frame batch. Pooled outputs favor retrieval and classification; dense token maps preserve more localization detail for detection or multimodal reasoning.
优势
- Reusable visual features
- Efficient classification, retrieval, and perception
- Can connect images to language models through a projector or cross-attention
权衡
- Resolution and patch size determine lost detail and token cost
- Training objectives create different blind spots
- A pooled vector is insufficient for many spatial tasks
适用场景
- The system must understand or retrieve visual content
- You can choose pooled versus spatial features based on the task
- Small text, charts, and domain imagery are evaluated explicitly
应避免或质疑的场景
- Visual generation is mistakenly expected from the encoder alone
- Input resolution removes required details
- A generic image benchmark substitutes for the actual domain
已发表的示例系列
- • Vision Transformer (ViT)
- • Convolutional and hierarchical vision backbones
- • The image tower in CLIP
常见组合
Contrastive / dual encodersMultimodal fusionImage and video generators