AI Inference Guide
AI Inference Overview
What is AI Inference?
AI inference is the process of using a trained machine learning model to make predictions or generate outputs from new input data. Unlike training, which requires massive computational resources, inference can be optimized for speed, efficiency, and deployment in various environments.
Edge & Device Inference
Privacy
Local execution can reduce data sent to third parties; verify telemetry, storage, and model behaviour before making compliance claims
Cost Profile
Shift compute to user hardware, while still budgeting for devices, electricity, support, and model distribution
Low Latency
Can avoid network round trips; measure local startup and generation time on target devices
Offline Capable
Can work offline once every required model/runtime asset is cached and remote fallbacks are disabled
Key Technologies
WebGPU
High-performance GPU acceleration directly in web browsers
WebAssembly (WASM)
Near-native performance for CPU computation in browsers
Model Quantization
Trades model size and memory use against task quality; measure the chosen method on your evaluation set
ONNX Runtime
Cross-platform inference with hardware-specific optimizations