Model Architectures
Decoder-only Transformers
Use a causal mask so each position sees only earlier tokens, matching the left-to-right generation process used by most general-purpose chat models.
Mental model
Compress the prefix into attention state, predict one continuation token, append it, and repeat.
Data flow
- Prefix tokens
- Causally masked self-attention
- Hidden state at each position
- Vocabulary logits
- Next token → repeat
How it trains
Next-token cross-entropy supplies a learning signal at every position. Post-training may add instruction examples, preferences, verifiable rewards, or tool trajectories.
How inference runs
Prefill processes the prompt; decode then emits tokens serially while the KV cache stores past attention keys and values. Context size, cache memory, and output length drive serving behavior.
Strengths
- One scalable interface for many generative tasks
- Strong in-context adaptation
- Can interleave prose, code, structured data, and tool-call tokens
Trade-offs
- Serial decode latency
- Quadratic prefill in standard full attention
- Prompt sensitivity and probabilistic output require validation
Use it when
- Open-ended generation or interactive chat is central
- Many tasks can share one model endpoint
- You can constrain and evaluate outputs in the product loop
Avoid or challenge it when
- A small encoder or deterministic component meets the requirement
- Guaranteed exact outputs are assumed without validation
- Long source corpora should be searched rather than placed wholesale in context
Illustrative published families
- • GPT-style causal language models
- • LLaMA research family
Commonly combines with
Text & chatMixture of ExpertsMultimodal fusionRAGTool systems