Service health
Request rate, queue time, time to first token, total latency, errors, retries, and saturation.
AI Inference Guide
AI Inference Monitoring
Monitor service health, model behavior, output quality, and spend together so that a healthy API cannot conceal a degraded product experience.
Four layers to observe
Model behavior
Model and prompt version, token counts, finish reason, tool calls, fallbacks, and context truncation.
Product quality
Task success, groundedness, safety checks, user feedback, and reviewed production samples.
Economics
Requests and tokens by feature, provider charges, cache effectiveness, quota use, and cost per successful task.
Alert on user impact
- Use latency and error objectives for each user-facing inference path.
- Alert on sustained changes and burn rate, not isolated noisy requests.
- Segment by model, provider, region, release, and feature before aggregating.
- Attach a runbook, owner, and rollback action to every paging alert.
Close the incident loop
- Confirm scope with traces and release metadata.
- Reduce impact using rollback, fallback, throttling, or feature controls.
- Preserve privacy-safe evidence for diagnosis and evaluation replay.
- Add the failure mode to tests, dashboards, and the operating runbook.