Static
Fixed-size batches that wait to fill. Fine for offline jobs, poor for interactive traffic because everyone waits for the batch.
AI Inference Guide
Serving Optimization
Systems-level techniques that decide how many requests a GPU fleet can serve and how fast: batching, KV-cache memory management, decode acceleration, phase separation, parallelism, and routing. Latency, throughput, and cost pull against each other; the workload decides which corner wins.
Batching: the first throughput lever
A single request leaves most of a GPU idle; grouping requests amortizes each weight read across many tokens. How you group them matters:
Dynamic
Batches close on a time window or size cap, bounding the wait. Better, but the batch still finishes at the pace of its slowest member.
Continuous
The scheduler works at token granularity: finished sequences leave mid-flight, queued requests join immediately. This is the mainstream default (vLLM, SGLang, TensorRT-LLM) and the single biggest utilization win.
Chunked prefill refines it further: long prompts are split into pieces interleaved with decode steps, so one huge prompt cannot stall token streaming for everyone else. Batching policy is a latency policy; measure TTFT and inter-token latency, not just tokens per second.
KV-cache memory: paging, reuse, offload
PagedAttention
Naive serving reserves contiguous KV memory for each request's maximum possible length, wasting most of it as fragmentation. PagedAttention borrows virtual memory ideas: the cache lives in fixed-size blocks allocated on demand, with a block table mapping each sequence's logical positions to physical blocks. Waste drops to roughly the last partial block, identical prefixes can share physical blocks, and the reclaimed memory becomes bigger effective batches. Introduced by vLLM and now standard across serving engines; it is a memory-management win rather than a faster kernel.
Prefix caching
If two requests start with byte-identical token prefixes (same system prompt, same examples, same chat history), the prefill work for that prefix can be computed once and reused, cutting TTFT and prefill compute on every hit. It requires exact token matches, which is why prompt discipline matters: static content first, volatile values (timestamps, user data, request IDs) as late as possible, deterministic serialization. Track the hit rate; a low one usually means the prompt template leaks variability into its head.
KV cache offloading
The cache grows linearly with context and concurrency, and GPU HBM runs out first. Offloading spills colder cache to CPU RAM, local SSD, or remote stores and pulls it back on demand, so idle chat sessions and long shared contexts stop pinning HBM (LMCache is the visible open-source example, integrated with vLLM). Worth it for long contexts and resumable sessions; the trade is transfer time against recompute, so benchmark both paths on your storage tiers.
Speculative decoding: faster tokens, same distribution
Decode is memory-bound, so there is spare compute at each step. A cheap drafter proposes several tokens; the target model verifies them in one pass, accepting the longest correct run. Accepted tokens arrive at a fraction of the cost; rejected drafts fall back to normal decoding, and exact verification schemes preserve the target model's output distribution.
- Drafters: a small separate model, extra decoding heads on the target (Medusa-style), feature-level drafting (EAGLE family), or simple n-gram lookup from the prompt. Implementations often draft several candidate continuations and verify the whole tree in one batched pass.
- The controlling variable is the acceptance rate: a well-aligned drafter on predictable text (code, structured output) helps a lot; a mismatched drafter can make serving slower.
- Interactions: the drafter consumes memory and scheduler complexity, and gains shrink at very high batch sizes where the compute slack disappears. Research directions are surveyed in advanced optimization.
Scaling out: disaggregation, parallelism, routing
Prefill/decode disaggregation
Prefill is compute-bound and bursty; decode is bandwidth-bound and steady. Colocated, they interfere: a long prefill stalls everyone's token stream. Disaggregation runs the phases on separate worker pools, scaled and tuned independently, with the prefill worker shipping the KV cache to a decode worker over a fast fabric. The transfer is the tax, so this pays off at fleet scale and hurts at small scale or for short prompts; frameworks include vLLM, SGLang, and NVIDIA Dynamo.
Parallelism strategies
Data parallelism replicates the model and splits traffic. Tensor parallelism shards individual layers across GPUs so oversized models fit, at the cost of communication on every step (it wants NVLink-class links). Pipeline parallelism assigns layer ranges to stages, with microbatching to fill the pipeline bubbles. Expert parallelism spreads MoE experts across devices. Real deployments compose these, and the right mix is empirical: tensor parallelism adds bandwidth pressure, while replication shrinks per-GPU KV headroom, so benchmark configurations rather than reasoning from first principles.
Inference routing
With many replicas, the router becomes an optimization surface. Round-robin ignores everything that matters for LLMs; better signals are active load, memory headroom, and above all cache locality: prefix-aware or session-sticky routing sends requests where their KV state already lives (consistent hashing on the prompt prefix is the common trick, used by projects like llm-d). Disaggregated fleets route by phase, and priority or SLA classes and adapter locality can feed the same scoring. This is distinct from model-selection routing across model tiers, covered in agentic patterns.
Offline batch inference
Not every workload is interactive. Embedding corpora, classification backfills, summarization sweeps, and evaluation runs tolerate hours of latency, which changes the optimization target entirely: saturate the hardware, use spare or off-peak capacity (providers sell discounted batch tiers for exactly this), and add post-processing and quality checks before results are consumed. Keeping batch work off the interactive fleet also protects your latency SLOs.
Benchmark before you believe
- Measure TTFT, inter-token latency, end-to-end latency (median and P99), throughput at fixed concurrency, and cost per request; single-number results conceal the trade-offs.
- Use your own traffic shape: prompt and output length distributions, concurrency pattern, cache-hit potential. Synthetic uniform workloads flatter every optimization.
- Load generators built for LLM serving (GenAI-Perf, LLMPerf, or k6/Locust scripts) report token-level metrics that generic HTTP tools miss.
- Record model, quantization, hardware, engine version, and configuration with every result; unlabeled benchmarks cannot be compared or reproduced.
- Quality benchmarks (MMLU-style suites) and performance benchmarks answer different questions; a serving change should be validated against both a latency target and an accuracy evaluation.