Архитектуры моделей
Encoder-only Transformers
Let every input token attend to tokens on both sides, producing contextual representations rather than an open-ended generation loop.
Ментальная модель
Read the entire input, contextualize each position, then pool or attach a task head.
Поток данных
- Token sequence
- Bidirectional self-attention blocks
- Contextual token states
- Pooler or task head
- Class, span, score, or vector
Как проходит обучение
Masked-token reconstruction is the classic pretraining objective, followed by task-specific classification, token labeling, span prediction, or metric learning.
Как выполняется инференс
One forward pass processes the available input. It is naturally suited to understanding and representation, but does not by itself generate an arbitrary continuation.
Сильные стороны
- Rich bidirectional context for classification and extraction
- Parallel processing across the input
- Often efficient for high-volume bounded tasks
Компромиссы
- Not an open-ended generator
- Maximum input length and quadratic attention can constrain long documents
- A task head or pooling strategy is usually required
Использовать, когда
- Classification, named entities, reranking, or embeddings dominate
- The complete input is available before prediction
- A smaller specialized model can meet the rubric
Избегать или пересмотреть, когда
- The primary output is long free-form text
- Streaming generation is required
- Inputs routinely exceed the evaluated context strategy
Примеры опубликованных семейств
- • BERT and RoBERTa research families
- • Encoder-based rerankers and classifiers
Часто сочетается с
Embedding modelsVision encodersContrastive / dual encodersRetrieval systems