模型架构
Decoder-only Transformers
Use a causal mask so each position sees only earlier tokens, matching the left-to-right generation process used by most general-purpose chat models.
思维模型
Compress the prefix into attention state, predict one continuation token, append it, and repeat.
数据流
- Prefix tokens
- Causally masked self-attention
- Hidden state at each position
- Vocabulary logits
- Next token → repeat
训练方式
Next-token cross-entropy supplies a learning signal at every position. Post-training may add instruction examples, preferences, verifiable rewards, or tool trajectories.
推理运行方式
Prefill processes the prompt; decode then emits tokens serially while the KV cache stores past attention keys and values. Context size, cache memory, and output length drive serving behavior.
优势
- One scalable interface for many generative tasks
- Strong in-context adaptation
- Can interleave prose, code, structured data, and tool-call tokens
权衡
- Serial decode latency
- Quadratic prefill in standard full attention
- Prompt sensitivity and probabilistic output require validation
适用场景
- Open-ended generation or interactive chat is central
- Many tasks can share one model endpoint
- You can constrain and evaluate outputs in the product loop
应避免或质疑的场景
- A small encoder or deterministic component meets the requirement
- Guaranteed exact outputs are assumed without validation
- Long source corpora should be searched rather than placed wholesale in context
已发表的示例系列
- • GPT-style causal language models
- • LLaMA research family
常见组合
Text & chatMixture of ExpertsMultimodal fusionRAGTool systems