モデルアーキテクチャ
Vision encoders
Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.
理解のためのモデル
A visual reader. It creates representations; generation requires a decoder or generative model.
データフロー
- Image or frames
- Patch, convolutional, or hybrid stem
- Vision backbone
- Spatial tokens / feature pyramid / pooled vector
- Task head or modality connector
学習方法
Supervised labels, masked-image modeling, self-distillation, reconstruction, and image–text contrastive objectives produce different visual invariances and levels of spatial detail.
推論の実行方法
The encoder runs once per image or frame batch. Pooled outputs favor retrieval and classification; dense token maps preserve more localization detail for detection or multimodal reasoning.
強み
- Reusable visual features
- Efficient classification, retrieval, and perception
- Can connect images to language models through a projector or cross-attention
トレードオフ
- Resolution and patch size determine lost detail and token cost
- Training objectives create different blind spots
- A pooled vector is insufficient for many spatial tasks
適する場合
- The system must understand or retrieve visual content
- You can choose pooled versus spatial features based on the task
- Small text, charts, and domain imagery are evaluated explicitly
避ける・再検討する場合
- Visual generation is mistakenly expected from the encoder alone
- Input resolution removes required details
- A generic image benchmark substitutes for the actual domain
公開された方式の例
- • Vision Transformer (ViT)
- • Convolutional and hierarchical vision backbones
- • The image tower in CLIP
よく組み合わせる要素
Contrastive / dual encodersMultimodal fusionImage and video generators