Sampling
temperature, top_p, top_k, and optionally a seed. Low temperature suits extraction, classification, and code; higher values suit ideation and creative text.
AI Inference Guide
Model Interaction & APIs
The contract between application and model: how requests are shaped, how outputs are constrained into machine-usable form, how models call tools, and the API surfaces that keep the whole thing portable.
Anatomy of a request
- Messages and roles: a system message sets standing instructions, user messages carry input, assistant messages carry prior model turns. The model sees the entire submitted list, not just the latest question.
- Templates and prefill: applications assemble prompts from templates with dynamic slots; pre-seeding the start of the assistant turn steers format and tone.
- Context management: the window is finite, so long conversations get truncated or summarized. What you drop changes the answer; make the policy deliberate.
- Serving impact: prompt structure is a performance decision. Keeping stable content (system prompt, examples) at the front and volatile content at the end is what makes prefix caching effective; see serving optimization.
Prompting technique itself (zero-shot, few-shot examples, chain-of-thought, role framing) is covered as a pattern catalog under patterns.
Inference parameters
Every request carries knobs that reshape the output distribution and the output contract:
Length and stopping
max_tokens caps generation and spend; stop sequences end it early. Both change results, not just cost.
Shaping and introspection
Repetition, presence, and frequency penalties; logit_bias to promote or ban tokens; logprobs for confidence signals; n or best_of for multiple candidates.
Parameter support and exact semantics differ per provider and runtime. How these settings interact with reproducibility is treated in depth in non-determinism.
Structured outputs
Pipelines need JSON, not prose. Two families of techniques get you there:
Validate and retry
Parse the response against a schema and re-prompt on failure (the Instructor pattern). Works with any API, including hosted ones, but failed attempts cost latency and tokens.
Constrained decoding
Mask invalid tokens at each step so only schema-conforming output can be produced (Outlines, XGrammar, Guidance; built into vLLM and SGLang, and exposed as JSON-schema modes by hosted providers). Guarantees shape, though a syntactically valid answer can still be semantically wrong.
A bare "JSON mode" without a schema only promises parseable JSON, not your JSON; prefer schema enforcement plus application-level validation of values.
Function calling and MCP
Function calling
You describe available tools (name, purpose, parameter schema); the model emits a structured call instead of prose; your code executes it and returns the result as a new message; the model continues with real data. It is next-token prediction producing a constrained format, so schema quality and tool descriptions drive reliability. Wrapped in a plan-execute-evaluate loop, this is the mechanism underneath agents; see agentic patterns. Support and quality vary across open models; verify with your runtime.
Model Context Protocol
MCP standardizes how assistants reach external systems: a host application runs MCP clients that connect to MCP servers, each exposing tools, data resources, and prompts over a common protocol, locally or remotely. Instead of bespoke integrations per app and per service, either side of the protocol is written once. Multiple servers can be attached to one host simultaneously.
Compatible API surfaces
Self-hosted engines and gateways imitate the major providers' HTTP APIs so existing SDKs work by changing only the base URL. This is what keeps applications portable across hosted and self-hosted backends.
OpenAI-compatible
- Chat completions and model-listing endpoints; streaming via a stream flag
- The de facto standard: vLLM, SGLang, and most gateways serve it
- Point the official SDK at your endpoint with a base URL override; the API key is often a placeholder for self-hosted servers
Anthropic-compatible
- Messages-style endpoint; auth via x-api-key plus an anthropic-version header
- System prompt is a top-level field, tool use arrives as typed content blocks
- Advanced features (prompt caching, extended thinking, image and document input) are only partially replicated by open backends
Compatibility is a spectrum, not a guarantee: parameter coverage and feature support differ per backend, so test the specific features you rely on before swapping endpoints.