Loading patterns…
Semantic Caching(SCA)
Cache LLM or agent responses keyed by embedding similarity rather than exact string match, so semantically equivalent queries reuse a stored answer. A similarity threshold decides a hit, a freshness TTL bounds staleness, and an invalidation policy evicts entries when source data changes. This cuts cost and latency on high-volume traffic and is distinct from prompt-prefix or KV caching.
In 30 seconds
- What
- Caches LLM responses by embedding similarity rather than exact text match, reusing answers when new queries exceed a similarity threshold.
- When to use
- High-volume query workloads where many semantically equivalent questions repeat and cost or latency reduction matters more than freshness.
- Watch out
- Similarity threshold tuning is brittle; too low causes stale answers on drifted topics, too high misses valid cache hits.
Ask the AI expert about this pattern
Opens the assistant with your question prefilled. You review it before sending.
Semantic Caching: Overview
Cache LLM or agent responses keyed by embedding similarity rather than exact string match, so semantically equivalent queries reuse a stored answer. A similarity threshold decides a hit, a freshness TTL bounds staleness, and an invalidation policy evicts entries when source data changes. This cuts cost and latency on high-volume traffic and is distinct from prompt-prefix or KV caching.
- Embedding-similarity cache keys
- Tunable similarity threshold for hits
- Freshness TTL to bound staleness
- Invalidation on source-data change
- Vector store backed lookup
- Cost and latency reduction at high volume
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.
References
The papers, specifications, and repositories this pattern is based on.
From the engineer behind this catalog
Get your agent architecture reviewed
This page documents one pattern. Your system runs dozens, and most failures live in how they fit together. Have the whole design reviewed against the 288 patterns in this catalog: architecture, reliability, evaluation and cost, every finding mapped to the pattern that fixes it.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September