Architectures de modèles
Decoder-only Transformers
Use a causal mask so each position sees only earlier tokens, matching the left-to-right generation process used by most general-purpose chat models.
Modèle mental
Compress the prefix into attention state, predict one continuation token, append it, and repeat.
Flux de données
- Prefix tokens
- Causally masked self-attention
- Hidden state at each position
- Vocabulary logits
- Next token → repeat
Entraînement
Next-token cross-entropy supplies a learning signal at every position. Post-training may add instruction examples, preferences, verifiable rewards, or tool trajectories.
Exécution de l’inférence
Prefill processes the prompt; decode then emits tokens serially while the KV cache stores past attention keys and values. Context size, cache memory, and output length drive serving behavior.
Atouts
- One scalable interface for many generative tasks
- Strong in-context adaptation
- Can interleave prose, code, structured data, and tool-call tokens
Compromis
- Serial decode latency
- Quadratic prefill in standard full attention
- Prompt sensitivity and probabilistic output require validation
À utiliser lorsque
- Open-ended generation or interactive chat is central
- Many tasks can share one model endpoint
- You can constrain and evaluate outputs in the product loop
À éviter ou remettre en question lorsque
- A small encoder or deterministic component meets the requirement
- Guaranteed exact outputs are assumed without validation
- Long source corpora should be searched rather than placed wholesale in context
Familles publiées à titre d’exemple
- • GPT-style causal language models
- • LLaMA research family
Souvent combinée avec
Text & chatMixture of ExpertsMultimodal fusionRAGTool systems