Semantic caching starts from a reasonable premise. When users ask the same question in different words, paying the model to regenerate an answer it has already produced is wasted spend. A semantic cache stores past responses and returns one whenever a new prompt is close enough in meaning, and reported hit rates near 70% make it one of the first cost and latency wins teams look for when the inference bill climbs.
The place to run it is the gateway layer. An AI gateway already sits between the application and every provider, so the request path where a cache pays off is the same one that handles routing, provider prompt caching, and per-request metrics. Pull semantic caching down to that layer and its hard parts, an embedding call on every request, a threshold that trades correctness for hit rate, and answers that go stale, become lifecycle concerns the gateway is built to manage rather than logic every team rebuilds per project. This guide covers what semantic caching is, how it differs from exact-match caching, where it wins and breaks, and how a gateway handles it for you.
Key takeaways:
Semantic caching embeds every incoming prompt and returns a stored response when vector similarity clears a threshold, skipping the model call on a hit.
The similarity threshold sets the hit rate and the false-positive rate at the same time, so no single value delivers both a high hit rate and few wrong answers.
A hit saves input and output tokens, but every request pays for an embedding call, so the economics only work at a sustained high hit rate.
Provider prompt caching and exact-match response caching operate at different layers and can run alongside a semantic cache.
Cached answers go stale when the underlying facts change, so a semantic cache needs invalidation or a TTL, not only a similarity threshold.
The gateway layer is where semantic caching belongs, running alongside provider prompt caching through
caching: 'auto', multi-provider routing, and per-request metrics.
Copy link to headingWhat is semantic caching for LLM apps?
Semantic caching is a technique that stores past prompt and response pairs and reuses a stored response when a new prompt is close enough in meaning to one already seen. Instead of matching request text exactly, it embeds each incoming prompt into a vector and searches for a near neighbor, so paraphrases of the same question can return the same cached answer.
The design pays its embedding cost up front and collects only when it hits. A cache is configured with a vector dimensionality, a maximum number of stored entries, and a distance threshold, and any stored embedding within that threshold of an incoming request vector counts as a cache hit. That threshold is the single most consequential setting in the system, which the next two sections work through in turn.
Copy link to headingHow semantic caching works
Every request follows the same path, whether it ends in a hit or a miss. The important detail is where the model call sits, because only a hit removes it.
The five steps run in order on each request:
Vectorize the incoming prompt with an embedding model such as
text-embedding-3-smallat 1,536 dimensions.Run an approximate nearest-neighbor search over the vector index.
Compare the nearest neighbor's similarity score to the threshold.
On a hit, return the stored response and make no model call.
On a miss, call the model, then store the new embedding and response.
A hit bypasses generation, so it saves both input and output tokens rather than input alone. Every request, including every miss, still pays for step one. Two implementation details decide whether the scores behave correctly at all. Normalize embeddings so the score behavior matches the index metric, and query with dot product on normalized vectors, where it equals cosine similarity but computes faster.
The upside, when it lands, is large. Cache-hit paths can complete in tens of milliseconds under favorable lookup conditions, and one benchmark returned in 25.3ms against roughly 7 seconds for a live model call. Whether those savings survive contact with real traffic is the threshold's problem, not the lookup's.
Copy link to headingSemantic caching vs. prompt caching vs. exact-match caching
Semantic caching is one of three caching layers available to an LLM app, and the three are often confused because they all reduce cost. They operate at different points in the request path and can run together, so the useful question is which layer to reach for first, not which one to pick.
The table below compares the three on the dimensions that determine cost and risk:
Provider prompt caching reduces prefill cost for a repeated, byte-identical prompt prefix, and the model still generates a fresh response. That makes it the right layer when queries are novel but share long static context, such as system instructions and tool schemas. Anthropic bills cached reads at a 90% discount to the input rate, and OpenAI discounts cached reads by 50 to 98.75% depending on the model. Exact-match caching sits higher up and returns a stored response for an identical request with no false-positive risk, while semantic caching is the only one of the three that trades correctness for a higher hit rate.
Copy link to headingWhere semantic caching breaks
A semantic cache breaks along two axes, and a gateway has to manage both. The similarity threshold governs the hit rate and the false-positive rate at the same time, so tuning for one tunes the other against you, and that tension shows up the moment a deployment leaves a demo. Separately, a cached answer that was correct when stored goes wrong when the underlying facts change, which no threshold value can detect.
Copy link to headingThe similarity threshold is a single knob
Set the threshold tight, toward 0.99, and only near-identical prompts match. The hit rate falls to low single digits and false positives become negligible, at which point the cache rarely fires while still paying an embedding call on every request. Loosen it toward 0.85 and the hit rate can climb to a large share of traffic, but false positives climb with it into the double digits, and each false positive is a confidently wrong answer served to a user.
No single value escapes the tradeoff, and more data does not tune it away. Conservative thresholds miss safe reuse, aggressive thresholds serve semantically incorrect responses, and the tradeoff is a property of the architecture rather than a tuning problem waiting for a better dataset.
Copy link to headingFailure modes that survive any threshold
Some failures are not about where the knob sits, because they come from how embeddings represent meaning. A threshold cannot separate cases the embedding places in the same region of vector space.
Three failure modes persist regardless of the threshold value:
Negation blindness: Embedding models place negated sentences close together, so "the theory is applicable" and "the theory is not applicable" register as highly similar despite meaning the opposite.
Parameter-identical framing: Prompts that share a sentence frame but differ in one operational parameter can embed above 0.95 because the shared frame dominates the vector, yet they require different answers.
Context blindness: A cached answer to "change the color to blue" after "draw a square" can match the cached answer for "change the color to red" after "draw a circle," even though it references a different prior state.
Each of these produces a hit the similarity score endorses and the answer contradicts. Tightening the threshold suppresses some of them at the cost of the hit rate, which returns the deployment to the same tradeoff.
Copy link to headingCached answers go stale
Even a correct hit has a shelf life. A cached answer about a price, a shipping status, a policy, or a document is right only until the underlying fact changes, and the similarity score keeps returning it long after it stops being true. Staleness is orthogonal to the threshold, because the prompt still matches perfectly, so no tuning of the similarity knob detects it.
Handling staleness is a second design requirement beyond hit rate, and it is a lifecycle concern rather than a matching one. A cache needs a time-to-live so entries expire on a schedule, and it needs targeted invalidation so a known change can evict the entries it affects. Both belong at the layer that owns the cache lifecycle, which is one reason a semantic cache is more reliable to run correctly at the gateway than scattered across application code.
Copy link to headingBad hits fail silently
A false positive is the worst kind of failure because it looks like success. The cache returns an incorrect answer with a 200 OK status, no exception, and no 5xx code, so nothing alerts. Without explicit instrumentation that samples cached responses against fresh generations, a team cannot tell whether the cache is working or quietly degrading.
The failure surface is not only accidental. An attacker who injects a crafted cache entry can redirect semantically similar queries to a response of their choosing, turning the same paraphrase weakness into an attack. A cache that returns the wrong answer on a benign rephrase can be steered to return a chosen one, which makes a semantic cache an input surface and not only a cost optimization.
Copy link to headingWhen a semantic cache earns its place
A semantic cache earns its place on one condition, and that condition is common enough to matter. Users independently phrase the same factual, context-independent question, and the per-request embedding cost stays small relative to the model call it avoids. Customer support and Q&A products often meet that bar, which is why the strongest published results come from there. The decision is quantitative, so the work is to measure the fraction of traffic that qualifies before committing to the pattern.
Copy link to headingMatch the cache to query repeatability
The one input that decides the outcome is how often different users ask the same thing in different words. High repetition across users is what makes reuse both safe and frequent, and it is the profile behind reported wins such as a patient-care voice app that reported a 70% hit rate and four times faster responses on Redis LangCache. Measure that fraction against the embedding overhead directly, rather than assuming the industry default applies to your workload.
Copy link to headingRule out agentic, personalized, and context-dependent traffic
Several traffic shapes look cacheable and are not, and they share one property. The prompt text underdetermines the correct answer. Recognizing them up front avoids shipping a cache that serves wrong answers at a high hit rate.
The following workloads are anti-patterns for semantic caching:
Agentic pipelines: In agentic pipelines the answer depends on prior tool results, so two identical-looking prompts can require different responses.
Personalized content: The same prompt has a different correct answer per user, which a shared cache cannot represent.
Code assistance: A one-word operand difference is semantically critical, exactly the distinction embeddings blur.
Retrieval-augmented generation: Retrieved documents change the effective query, so the visible prompt is not the full input.
For any of these, a higher hit rate means more contextually wrong answers with no error signal, which is the opposite of the intended result.
Copy link to headingInstrument for false positives before shipping
A semantic cache cannot be shipped on hit rate alone, because hit rate says nothing about how many of those hits were wrong. Before it reaches production, sample cached responses against fresh generations and measure the false-positive rate at the chosen threshold, then decide whether that error rate is acceptable for the product. A cache without that measurement is running blind on the one metric that determines whether it helps or harms.
Copy link to headingHow an AI gateway handles semantic caching for you
A semantic cache is not a standalone service bolted onto an app. It is one component of gateway architecture, sharing the request path with routing, provider prompt caching, and per-request metrics. Running it at the gateway layer means the surrounding lifecycle is handled for you, and the semantic layer composes on top of primitives that the gateway already exposes. AI Gateway handles the native caching, wraps the cache in reliability and observability, and gives you the hook, then a Marketplace vector store, such as Upstash Vector, provides the semantic layer.
Copy link to headingProvider prompt caching with one setting
Prompt caching is the first caching layer the gateway gives you, and it applies to the common case of novel queries that share a long static prefix, such as system instructions and tool schemas. It reduces prefill cost without any of the correctness risk a semantic cache carries, so it is worth turning on before anything else.
On AI Gateway, this is a single setting:
const result = streamText({ model: 'anthropic/claude-sonnet-4.6', providerOptions: { gateway: { caching: 'auto' } }, messages,});
With caching: 'auto', AI Gateway applies the provider's prompt-caching strategy where supported, including passing Anthropic cache-control markers automatically, and surfaces the resulting metrics. It carries zero false-positive risk because the model still generates a fresh response, and it composes with any exact-match or semantic cache a team assembles above it.
Copy link to headingReliability and observability around the cache
Production traffic keeps drifting toward shapes a cache cannot serve. Tool-call requests carried 58.9% of all tokens through AI Gateway by April 2026, up from 31.6% six months earlier, according to the AI Gateway production index. Agentic traffic is context-sensitive by definition, so similarity between two prompts says little about whether they deserve the same answer, which is exactly the traffic a semantic cache should not try to serve.
The gateway owns that remainder so the cache does not have to. AI Gateway runs multi-provider fallback on Fluid compute across 126 Points of Presence, and fallback routing rescues 3.5% of requests and 5.1% of tokens that would otherwise have returned errors, with over one trillion tokens recovered per month. The same layer records the model, tokens, and cost of every request, which is how a team samples cached responses for false positives and staleness instead of shipping the cache blind. The cache handles the repetitive slice of traffic, and the gateway keeps the rest reliable and observable.
Copy link to headingAdd the semantic layer with AI SDK middleware
The gateway exposes the hook for the semantic layer through AI SDK language model middleware, so a cache slots into the same request path without changing call sites. This is where the semantic layer lives at the gateway layer rather than as a separate service.
The recommended pattern pairs wrapLanguageModel with simulateReadableStream to replay a cached response as a stream:
import { wrapLanguageModel, type LanguageModelV4Middleware } from 'ai';
const cacheMiddleware: LanguageModelV4Middleware = { wrapGenerate: async ({ doGenerate, params }) => { // look up params in your vector store, return the hit or fall through return doGenerate(); }, wrapStream: async ({ doStream }) => doStream(),};
const model = wrapLanguageModel({ model: 'anthropic/claude-sonnet-4.6', middleware: cacheMiddleware });
LanguageModelV4Middleware is the AI SDK 7 middleware type, and it exposes wrapGenerate and wrapStream for the non-streaming and streaming paths. One constraint is structural. An in-memory Map will not survive across Vercel Function invocations, so the store has to be external. The gateway gives you the hook at the right layer, and you supply the vector store and the threshold that fit your workload.
Copy link to headingServerless-native vector storage from the Marketplace
The external store is the other half of the semantic layer, and a serverless deployment model needs a store that matches it rather than a server that a team has to run. Upstash Vector from the Vercel Marketplace fits that model with a TypeScript SDK, REST configuration, and unified billing, and Upstash Semantic Cache wraps it in a fuzzy key-value API with a configurable minProximity. Setting a time-to-live on cached entries is where the staleness problem gets handled, so answers expire on a schedule instead of outliving the facts they encode.
The managed options differ mainly in language support and maturity, which the table below compares:
For TypeScript teams on a serverless model, Upstash is the closest fit, while GPTCache's Python-only design and open security issues make it a poor choice for new production work. Whichever store a team picks, the correctness and instrumentation work from the previous sections still applies.
Copy link to headingRun semantic caching at the gateway layer
Whether a semantic cache pays off still comes down to one measurement, the fraction of traffic made up of semantically equivalent questions from different users, weighed against the cost of embedding every request. A reported 70% hit rate on a support workload says nothing about an agentic or personalized workload, where the same headline number would mostly count wrong answers. Measure the fraction first, then run the cache where its hard parts are already handled.
The gateway layer is that place, and it supplies each piece the pattern needs:
Provider prompt caching, one setting:
caching: 'auto'on AI Gateway applies the provider's caching strategy and carries zero false-positive risk.Routing and fallback around the cache: Multi-provider fallback on Fluid compute keeps the requests a cache cannot serve from turning into errors.
Per-request metrics to catch bad hits: Gateway observability is how you sample cached responses for false positives and staleness rather than shipping blind.
The AI SDK middleware hook:
wrapLanguageModelwithLanguageModelV4Middlewareadds the semantic layer without changing call sites.A serverless-native store from the Marketplace: Upstash Vector and Semantic Cache hold the embeddings and expire entries on a TTL before they go stale.
Start a new project to wire caching, routing, and metrics in from the beginning, or browse Vercel templates for AI patterns already set up for production.
Copy link to headingFAQs about semantic caching
Copy link to headingDoes Vercel AI Gateway support semantic caching?
AI Gateway handles prompt caching natively through caching: 'auto' with providers such as Anthropic and MiniMax, and it gives you the gateway-layer hooks to add semantic caching, AI SDK middleware plus a vector store, so the semantic layer runs at the same layer as routing, fallback, and metrics rather than as a separate service.
Copy link to headingWhat similarity threshold should I start with?
Start at 0.95, then sample production traffic for false positives and adjust to your error tolerance. Documented defaults range from a GPTCache default of 0.75 to Upstash examples at 0.95, and practitioner guidance often sits around 0.92 to 0.97.
Copy link to headingHow is semantic caching different from provider prompt caching?
Provider prompt caching saves prefill compute on a repeated prompt prefix while the model still generates a new response, so it carries no false-positive risk. A semantic cache hit skips generation entirely and saves input and output tokens, but it adds an embedding call to every request and can return a wrong answer above its threshold.
Copy link to headingCan semantic caching be used in Next.js App Router applications?
Yes, through the AI SDK middleware with LanguageModelV4Middleware paired with an external vector store. An in-memory Map will not persist across Vercel Function invocations, so a serverless-compatible store such as Upstash Vector is required for the cache to survive between requests.
Copy link to headingWhat workloads should avoid semantic caching?
Avoid it on agentic pipelines where responses depend on prior tool outputs, personalized content where the same prompt has a different correct answer per user, and code tasks where a single operand changes the meaning. In those cases, a cache returns contextually wrong answers with no error signal.