Training
- Updates weights via backpropagation over batches of data
- Runs hours to weeks on multi-node clusters
- Tooling such as PyTorch, JAX, DeepSpeed, Megatron
- Only worth doing yourself when no existing model fits
AI Inference Guide
Inference Fundamentals
What actually happens between a prompt and a response: the generation loop, the two phases that dominate latency, the hardware the work lands on, and the metrics that describe it.
Training and inference are different jobs
Training adjusts model weights over a large dataset, typically once, on a large accelerator cluster. Inference applies the frozen weights to new inputs, continuously, for every request your product serves. Training cost is mostly a one-time investment; inference cost compounds with traffic, which is why serving efficiency decides the economics of an AI product.
Inference
- Weights stay fixed; each request is a forward pass
- Latency measured in milliseconds to seconds, per request
- Serving stacks such as vLLM, SGLang, TensorRT-LLM, LMDeploy
- Cost scales with traffic and grows for the life of the product
A related distinction: an inference framework executes the model efficiently, while an inference server wraps it with batching, queuing, streaming, health checks, autoscaling, and an HTTP or gRPC API. Production systems need both.
How a model turns a prompt into text
- 1. TokenizationThe prompt is split into tokens and mapped to IDs from a fixed vocabulary. Token count, not character count, drives cost and context limits.
- 2. EmbeddingToken IDs become vectors; positional information (for example rotary embeddings) tells the model where each token sits in the sequence.
- 3. PrefillThe whole prompt is processed in one parallel pass through the transformer layers. Self-attention compares every token with every earlier token; a causal mask stops tokens from seeing what comes after them. This phase is compute-bound and determines time to first token.
- 4. DecodeTokens are generated one at a time. Each step reads the model weights and the cached attention state, so this phase is memory-bandwidth-bound and determines inter-token latency.
- 5. KV cacheAttention keys and values computed for earlier tokens are cached so each new token only attends to stored state instead of recomputing the whole sequence. The cache grows linearly with context length and competes with weights for GPU memory.
- 6. Sampling and streamingA next token is chosen from the probability distribution (greedy, temperature, top-k, top-p), appended, and usually streamed to the client until a stop condition or token limit is hit.
The prefill/decode split explains most serving behavior: long prompts hurt time to first token, long outputs hurt total latency, and almost every optimization on the serving optimization page targets one phase or the other. Token-by-token generation is the dominant paradigm but not the only one: diffusion-based language models generate whole passages through iterative refinement instead, an active research direction with different latency behavior.
Where inference runs
CPU
General-purpose, available everywhere, limited parallel throughput. Viable for small or heavily quantized models, prototyping, and edge boxes without accelerators.
GPU
The default for LLM serving: thousands of parallel cores and high-bandwidth memory (HBM) matched to the matrix multiplications that dominate the workload. Ecosystems: CUDA, ROCm.
TPU and other ASICs
Purpose-built tensor accelerators (for example TPUs with XLA/JAX) can be efficient at scale, at the price of a narrower software ecosystem and provider coupling.
Selection criteria are model size, request volume, latency targets, cost, and the infrastructure you already operate. Browser, edge, and mobile execution are covered in web inference and edge and mobile.
The metrics that describe an inference service
| Metric | Meaning | When it matters most |
|---|---|---|
| TTFT | Time to first token, dominated by queueing plus prefill | Chat and any interactive UI |
| ITL / TPOT | Latency between tokens; time per output token averages it over the response | Perceived streaming speed |
| E2EL | End-to-end request latency; TPOT is roughly (E2EL - TTFT) / (output tokens - 1) | Batch steps and tool-call chains |
| TPS / RPS | Output tokens per second and requests per second at a given concurrency | Capacity and cost planning |
| Goodput | Only the requests that met their latency objective | SLO-bound production services |
- Report median and tail (P95/P99) values; averages hide the slow requests users remember.
- Latency and throughput trade against each other, and batch size is the main lever between them.
- Always state model, hardware, sequence lengths, and concurrency next to any number; metrics are meaningless without the workload.