Model Architectures
Vision encoders
Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.
Mental model
A visual reader. It creates representations; generation requires a decoder or generative model.
Data flow
- Image or frames
- Patch, convolutional, or hybrid stem
- Vision backbone
- Spatial tokens / feature pyramid / pooled vector
- Task head or modality connector
How it trains
Supervised labels, masked-image modeling, self-distillation, reconstruction, and image–text contrastive objectives produce different visual invariances and levels of spatial detail.
How inference runs
The encoder runs once per image or frame batch. Pooled outputs favor retrieval and classification; dense token maps preserve more localization detail for detection or multimodal reasoning.
Strengths
- Reusable visual features
- Efficient classification, retrieval, and perception
- Can connect images to language models through a projector or cross-attention
Trade-offs
- Resolution and patch size determine lost detail and token cost
- Training objectives create different blind spots
- A pooled vector is insufficient for many spatial tasks
Use it when
- The system must understand or retrieve visual content
- You can choose pooled versus spatial features based on the task
- Small text, charts, and domain imagery are evaluated explicitly
Avoid or challenge it when
- Visual generation is mistakenly expected from the encoder alone
- Input resolution removes required details
- A generic image benchmark substitutes for the actual domain
Illustrative published families
- • Vision Transformer (ViT)
- • Convolutional and hierarchical vision backbones
- • The image tower in CLIP
Commonly combines with
Contrastive / dual encodersMultimodal fusionImage and video generators