Serverless API
A provider hosts the model and bills per token. Fastest start, zero GPU operations, elastic by default. You accept their models, limits, data terms, and price curve; at sustained volume it is usually the most expensive path.
AI Inference Guide
Capacity Planning & Hosting Choices
The decisions to make before anything ships: how the model is hosted, which model and GPU fit the workload, whether the memory math closes, and which serving framework runs it.
The hosting spectrum
Self-hosted cloud
You rent GPUs and run a serving stack (vLLM, SGLang, TensorRT-LLM). Full control over models, versions, and optimizations such as prefix caching or disaggregation, in exchange for real DevOps load: capacity, upgrades, and incident response are yours.
Bring Your Own Cloud
A vendor operates its serving platform inside your cloud account: their control plane, your data plane. Keeps data in your VPC for sovereignty and compliance, uses your credits and committed-use discounts, and reduces egress, while outsourcing most operations.
On-prem
Your hardware, your data center. Strongest data control and predictable capex; suits steady, high-volume, or regulated workloads. Costs are upfront GPU investment, procurement lead times, hardware aging, and the ops and talent to run it.
Decide on four axes: data privacy and compliance, cost at your actual scale, need for customization (fine-tuned weights, serving-level optimizations), and the operations capacity of your team. Compare options with measured TTFT, throughput, and cost per successful task, not with list prices; the providers page has an evaluation checklist.
Choosing the model
- Variant: base models are for further training; instruction-tuned or chat variants are what you serve. Small language models handle narrow tasks at a fraction of the cost.
- Architecture: dense models activate every weight per token; mixture-of-experts models activate a subset, trading lower compute per token for a larger memory footprint and more complex serving.
- Beyond text: embedding models (retrieval, RAG), rerankers, vision-language models, speech and image models; many products chain several of these per request.
- Weight formats: safetensors is the standard for GPU serving; GGUF targets llama.cpp-style local and CPU runtimes, usually pre-quantized (for example Q4/Q5/Q8 levels).
- Sourcing: model hubs (Hugging Face, ModelScope) distribute weights, often behind licence gates that require authenticated download; read the licence before it reaches production.
Choosing the GPU
What to compare
- VRAM: the hard ceiling for weights plus KV cache plus overhead
- Memory bandwidth: governs decode speed, the memory-bound phase
- Compute (tensor-core FLOPS): governs prefill and batch throughput
- Native precision support: FP8 and newer formats need recent generations
- Interconnect: NVLink-class links inside a node, InfiniBand or RoCE across nodes, if you shard models
Practical constraints
- Consumer cards suit development; data-center parts add VRAM, bandwidth, and multi-GPU links
- Software ecosystem: CUDA has the deepest tooling; ROCm support varies by project and version
- Control, availability, and price pull against each other; the cheapest SKU you cannot obtain is not an option
- If the model does not fit, the levers are quantization, tensor or pipeline parallelism, and KV offloading, each with costs of its own
Does the memory math close?
Estimate before renting anything. Three consumers share GPU memory:
- 1. WeightsParameters x bytes per parameter: roughly 2 bytes at FP16/BF16, 1 at FP8/INT8, 0.5 at INT4. An 8B model is about 16 GB at FP16 and about 4.5 GB at INT4. Add on the order of 10-20 percent for runtime overhead.
- 2. KV cachePer token, roughly 2 x layers x KV heads x head dimension x bytes per value; multiply by tokens in flight across every concurrent sequence. This is what actually limits concurrency and context length on a given card, and why long-context serving is a memory problem before it is a compute problem.
- 3. Activations and workspaceIntermediate buffers, CUDA graphs, and framework reservations claim the rest. Leave explicit headroom; an allocator at 99 percent utilization fails on the first long request.
Then validate with a load test at your real prompt and output lengths, since fragmentation and scheduler behavior move the practical limit below the arithmetic one.
Choosing the serving framework
The framework owns batching, KV cache management, streaming, and multi-GPU execution, and most expose an OpenAI-compatible API so applications stay portable.
Production GPU serving
vLLM and SGLang are the mainstream open-source engines (continuous batching, PagedAttention-style memory management, structured output support). TensorRT-LLM extracts maximum performance on NVIDIA hardware at higher integration cost; LMDeploy emphasizes quantized, high-concurrency serving.
Local and edge
llama.cpp (and Ollama on top of it) covers CPU and single-machine use with GGUF quantization; MLC-LLM compiles models for phones and browsers. These prioritize footprint and portability over aggregate throughput.
- Check model-architecture support first; a faster engine that cannot run your model is irrelevant.
- Decide library mode (embedded in your process) versus server mode (a separate service); production almost always wants server mode.
- Benchmark the shortlist on your own workload; ranking flips with model, hardware, and sequence shape. See the comparison on the libraries page.