AI Inference Guide
Why Inference is Non-Deterministic
The Myth of the "Same" Question
LLM outputs can vary even when two requests look identical. Sampling deliberately introduces variation; serving changes, hidden context, tool results, and some parallel kernels can add more. Byte-identical reproducibility requires control over the full input, decoding configuration, model/runtime version, and execution environment.
Key Insight: The variations arise from four primary domains: Generation & Sampling, Context & Input, System & Architecture, and Hardware & Implementation. Understanding these factors is crucial for building reliable AI systems.
Category 1: Intentional Generation & Sampling Parameters
The "Dice Roll": sampling can increase output diversity
1. Temperature
Rescales token logits before sampling, changing how concentrated the probability distribution is.
2. The Seed
Computers use a "seed" (starting number) to generate pseudo-random sequences.
3. Top-P (Nucleus Sampling)
The model samples from the smallest token set whose cumulative probability reaches the threshold (for example, 0.9). The candidate set is recomputed at each generation step.
4. Top-K
The model samples from the K most probable tokens (for example, the top 50).
5. Alternative Sampling Strategies
Samplers such as Mirostat or typical sampling use different rules to select candidate tokens.
Category 2: Output-Shaping Request Configuration
Changing these request settings changes the output contract; it is not evidence of nondeterminism
6. Repetition/Frequency/Presence Penalties
These settings lower scores for tokens that have already appeared and can discourage repetition. Changing a penalty changes the candidate distribution even when the seed is fixed.
7. Stopping Conditions (Max Tokens)
Hard limit on response length. A run with max_tokens=50 will differ from max_tokens=500.
8. Stopping Conditions (Stop Sequences)
The model stops if it generates a configured string (for example, "\nHuman:"). With stochastic sampling, different paths may encounter a stop sequence at different points.
Category 3: Contextual & Input-Time Variables
The "Memory": model input often includes more than the latest user message
9. Chat History (The Context Window)
Most common reason for perceived non-determinism. Model sees entire preceding conversation, not just last question.
A single changed token creates a different input and may change the output
10. Context Window Limit (The "Forgetting" Point)
Models have finite context windows whose size varies by model. Long conversations may be truncated, summarized, or rejected by the surrounding application; content outside the submitted context is unavailable to the model.
11. System Prompt
A higher-priority instruction supplied by the application or API. It may be visible to the developer but hidden from an end user, and changes to it can alter response style and behaviour.
12. Retrieval and External Tools
A major source of request-to-request input variation. In a search-grounded workflow, the application may:
- Pauses generation
- Calls external tool (e.g., Google Search API)
- Receives new, dynamic data (search results)
- Injects this data into context
Tool results can change between requests, which changes the model input and may change the answer
Category 4: System & Architectural Variables
The "Orchestra" - The "model" is often a complex system of multiple models
13. Model Updates & Versions
The model you use today may be a new, retrained, or updated version from yesterday. v1.2 model will respond differently than v1.3 model.
14. A/B Testing & Canary Rollouts
A provider or your own gateway may route traffic between versions during a rollout. If version pinning is unavailable, two requests can reach different deployments and produce different answers.
15. Speculative Decoding
A draft model proposes tokens that the target model verifies. Exact implementations aim to preserve the target distribution, but runtime, batching, approximation, or configuration differences can still affect reproducibility.
16. Mixture of Experts (MoE) Architecture
A router selects a subset of experts for each token. The routing function can be deterministic for a fixed input, while capacity limits, batching, and implementation details may create deployment-level variation.
Category 5: Hardware & Low-Level Implementation
The "Physics" - Deepest and most difficult-to-control source of variation
Caution: Some runtime choices can vary even when temperature is 0 and a seed is fixed.
17. Parallel Kernels and Floating-Point Reduction Order
Fixed operations in a fixed order can be repeatable. Variation can arise when parallel kernels or reductions execute in a different order and rounding differences propagate:
- Parallelism: GPUs perform billions of calculations in parallel
- Floating-Point Arithmetic: Computer math with decimals is not perfectly associative
- Consequence: Small score changes can alter token selection when candidates are very close
Example: (a+b)+c ≠ a+(b+c) due to rounding → 0.81234567 vs 0.81234568 → Different word chosen
18. Quantization Configuration
A quantized model uses lower-precision representations than another build and can therefore produce different scores or outputs. A fixed quantized model/runtime is not inherently nondeterministic; record the exact quantization and runtime as part of the model version.
19. Batching
Your request is processed in a "batch" with other users' requests. The way these are grouped and padded can change exact order of GPU calculations, triggering floating-point issues.
20. Inference-Time Dropout
Dropout is normally disabled for inference. Research techniques such as Monte Carlo dropout deliberately keep it active; a production runtime configured that way will be stochastic unless its random state is controlled.
Key Takeaways for Builders
Achieving Near Determinism
- Set temperature to 0 (greedy decoding)
- Use a fixed seed value
- Control all input context exactly
- Use same model version
- Accept that hardware-level variations may still occur
Embracing Non-Determinism
- Build systems that handle variation gracefully
- Use evaluation metrics that account for semantic equivalence
- Use bounded retries only for safe or idempotent operations
- Focus on consistency of quality, not exact outputs