Modellarchitekturen

Backbone · bidirectional context

Encoder-only Transformers

Let every input token attend to tokens on both sides, producing contextual representations rather than an open-ended generation loop.

Denkmodell

Read the entire input, contextualize each position, then pool or attach a task head.

Datenfluss

Token sequence
Bidirectional self-attention blocks
Contextual token states
Pooler or task head
Class, span, score, or vector

So wird trainiert

Masked-token reconstruction is the classic pretraining objective, followed by task-specific classification, token labeling, span prediction, or metric learning.

So läuft die Inferenz

One forward pass processes the available input. It is naturally suited to understanding and representation, but does not by itself generate an arbitrary continuation.

Stärken

Rich bidirectional context for classification and extraction
Parallel processing across the input
Often efficient for high-volume bounded tasks

Zielkonflikte

Not an open-ended generator
Maximum input length and quadratic attention can constrain long documents
A task head or pooling strategy is usually required

Geeignet, wenn

Classification, named entities, reranking, or embeddings dominate
The complete input is available before prediction
A smaller specialized model can meet the rubric

Vermeiden oder hinterfragen, wenn

The primary output is long free-form text
Streaming generation is required
Inputs routinely exceed the evaluated context strategy

Beispielhafte veröffentlichte Familien

• BERT and RoBERTa research families
• Encoder-based rerankers and classifiers

Häufig kombiniert mit

Embedding modelsVision encodersContrastive / dual encodersRetrieval systems

Primärquellen

BERT