Архитектуры моделей
Vision encoders
Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.
Ментальная модель
A visual reader. It creates representations; generation requires a decoder or generative model.
Поток данных
- Image or frames
- Patch, convolutional, or hybrid stem
- Vision backbone
- Spatial tokens / feature pyramid / pooled vector
- Task head or modality connector
Как проходит обучение
Supervised labels, masked-image modeling, self-distillation, reconstruction, and image–text contrastive objectives produce different visual invariances and levels of spatial detail.
Как выполняется инференс
The encoder runs once per image or frame batch. Pooled outputs favor retrieval and classification; dense token maps preserve more localization detail for detection or multimodal reasoning.
Сильные стороны
- Reusable visual features
- Efficient classification, retrieval, and perception
- Can connect images to language models through a projector or cross-attention
Компромиссы
- Resolution and patch size determine lost detail and token cost
- Training objectives create different blind spots
- A pooled vector is insufficient for many spatial tasks
Использовать, когда
- The system must understand or retrieve visual content
- You can choose pooled versus spatial features based on the task
- Small text, charts, and domain imagery are evaluated explicitly
Избегать или пересмотреть, когда
- Visual generation is mistakenly expected from the encoder alone
- Input resolution removes required details
- A generic image benchmark substitutes for the actual domain
Примеры опубликованных семейств
- • Vision Transformer (ViT)
- • Convolutional and hierarchical vision backbones
- • The image tower in CLIP
Часто сочетается с
Contrastive / dual encodersMultimodal fusionImage and video generators