Prompt caching across providers: what's automatic, what's explicit, and what it actually saves

Prompt caching questions usually start with an engineer staring at a log where cache_read_input_tokens is zero on every request, even with caching switched on.

Read discounts have converged near 90% across providers, so the number that decides your bill is no longer the discount but prefix stability. One request-specific field before the cache boundary drops the hit rate to zero on every provider.

This guide covers what each provider automates, what it makes explicit, and what prompt caching actually saves, along with the prompt-architecture fixes that keep hit rates high.

Key takeaways:

• Prompt caching is exact prefix reuse of key-value (KV) tensors at the inference layer. One request-specific field before the cache boundary drops the hit rate to zero on every provider.

• Read discounts have converged near 90%, but write premiums still shape the bill. With Anthropic at 1.25× to 2× and OpenAI GPT-5.6 at 1.25×, one cache read covers the 5-minute write premium and two cover the 1-hour premium.

• AI Gateway's caching: 'auto' applies cache markers for you, at the default 5-minute time-to-live (TTL). Workloads that depend on Anthropic's 1-hour TTL should set markers manually or call the provider directly.

• Tool-call requests now carry 58.9% of tokens through AI Gateway, up from 31.6% in October 2025. Agentic workloads offer the largest caching opportunity and the easiest way to break a prefix.

• A hit rate below 60% on a stable-prefix workload is a prompt architecture problem, not a provider problem. The fix is almost always prompt structure, not a provider parameter.

Copy link to headingWhat is prompt caching?

Prompt caching is the reuse of computed key-value (KV) tensors for an identical prompt prefix across separate requests. When a new request begins with the same token sequence as an earlier one, the inference engine skips the prefill step for that prefix and reads the stored tensors instead of recomputing them. The model still runs for everything after the matching prefix, so only the shared, stable portion is served from cache.

Because the match is on exact tokens, prompt caching rewards stable prompt architecture and punishes anything request-specific placed early. A cached read is billed at a fraction of the base input rate, so a large system prompt that would otherwise be reprocessed on every call becomes near-free after the first write. Exact prefix reuse is the single property that makes prompt caching the highest-impact cost lever for agentic and multi-turn workloads.

Copy link to headingHow prompt caching differs from semantic and response caching

"Caching" means three different things in LLM inference, and conflating them sends optimization effort to the wrong layer. Prompt caching reuses the KV tensors for a shared prefix and always returns a freshly generated response. The two patterns it gets confused with behave differently:

• Semantic caching: An embedding-similarity match returns a stored response when a new query is close enough to a previous one, so the model never runs. It saves output tokens but carries a correctness risk set by the similarity threshold.

• Response caching: An exact-match hash lookup returns a stored response for an identical request, consuming no tokens at all.

Prompt caching sits between them. It removes prefill cost for the matching prefix while the model still generates every output token, which is why it never changes the answer a request would have produced.

Copy link to headingWhat prompt caching does at the GPU level

Transformers compute key and value tensors for every input token during the prefill phase. Without prefix caching, a 10,000-token system prompt is recomputed from scratch on every request, even when it has not changed. With it, the engine stores those tensors and skips prefill for any request that starts with an identical token sequence, then runs the model normally for the new tokens that follow.

The catch lives in how the tensors are built. The KV tensor at position N depends on that token and every token before it, so one changed token at position K invalidates every cached block from K onward. Serving engines like vLLM go further and quantize hits to fixed block boundaries, where only full, hash-matched blocks reuse cached state. That is why a prompt that is almost identical never partially hits. The prefix either matches to the block or it does not.

Copy link to headingHow providers differ on prompt caching activation and cost

Headline read discounts have partly converged, which moves the real decision to four other dimensions. What matters now is how caching is activated, what a write costs, how long an entry lives, and how many tokens a prefix needs before it caches at all. The activation model matters most in production, because a 90% discount you cannot reliably trigger is worth less than a 50% discount that fires on every request.

Across the dimensions that decide a production bill, the six providers line up like this:

OpenAI's writes are no longer free. From GPT-5.6 on, a cache write bills at 1.25× the uncached input rate, so the assumption that OpenAI caching costs nothing to populate no longer holds. And Anthropic's minimum threshold now varies widely by model, which quietly decides whether short prompts cache at all.

Copy link to headingAutomatic, explicit, and silently broken activation

Providers split into two camps on activation. OpenAI models before GPT-5.6, Google Gemini, DeepSeek, and Groq cache automatically with no code changes, while Anthropic, Bedrock Claude, and OpenAI from GPT-5.6 on require explicit cache markers in the request body. GPT-5.6 moved OpenAI into the explicit camp, adding prompt_cache_key and developer-placed breakpoints that pull its caching model closer to Anthropic's.

Automatic does not mean guaranteed. A hit still depends on the request landing on a server that already holds the prefix, and OpenAI community reports document hit rates near 50% even on immediate resends. The practical takeaway is to pick the activation model you can actually satisfy. Groq's automatic 50% discount can beat Anthropic's 90% if your pipeline cannot hold a byte-identical prefix.

Copy link to headingRead discounts have converged near 90%

On reads, the field has bunched up. Anthropic bills a cache read at 0.1× the base input rate, a 90% discount, and Gemini's implicit caching on Vertex lands at the same 90%. OpenAI reaches up to 90% on GPT-5.6, though older generations sit lower, historically around 50% on the GPT-4o family and roughly 75% on GPT-5.5. DeepSeek is the outlier on the cheap end, with cache reads on its current models billed near 98% below the miss rate.

Because reads are so close across providers, the read discount is rarely where a caching decision is won or lost anymore. Write premiums and TTL behavior are.

Copy link to headingWrite premiums and the break-even point

A cache write is not free on every provider, and the premium is the variable that decides whether caching pays off. Anthropic charges 1.25× the base input rate for a 5-minute entry and 2× for a 1-hour entry, and OpenAI charges 1.25× on GPT-5.6. Gemini, DeepSeek, and Groq carry no per-write premium on their automatic paths, though Gemini's explicit caches add a storage fee per token-hour.

The break-even is straightforward once you separate the premium from the base cost you would have paid anyway. The general point is h* = (w − 1) / (1 − r), where w is the write multiplier and r is the read multiplier. At Anthropic's 5-minute rates (w = 1.25, r = 0.10), that is about 0.28 subsequent reads, which rounds to a single cache hit. The 1-hour tier at 2× writes breaks even at about 1.11 subsequent reads, so two hits. For a system prompt sent on every request, payback is effectively instant, and the multipliers behind this math are the ones Anthropic lists in its prompt caching documentation. For a document referenced only a few times, do the arithmetic before you cache it.

Copy link to headingTTLs and minimum-token floors

On TTL, Anthropic defaults to 5 minutes and refreshes on every hit, with a 1-hour option at the higher write premium. OpenAI's GPT-5.6 guarantees a 30-minute minimum, Gemini's implicit cache offers no guaranteed lifetime, and Groq holds entries for roughly two hours.

On minimum prefix length, Anthropic's floor now depends on the model. Claude Opus 5, Claude Sonnet 5, and the Mythos-class Fable 5 cache prefixes as short as 512 to 1,024 tokens, while Claude Haiku 4.5 needs 4,096. A prompt under the floor produces no cache write and no error message, so a short system prompt can look configured while caching nothing. OpenAI sits at 1,024 tokens and Gemini around 1,024 to 2,048 by model. Check the floor for the exact model you call rather than assuming one number carries across a family.

Copy link to headingHow to keep your prompt cache hit rate high

The provider you pick sets the ceiling on savings. Prompt structure decides how much of that ceiling you reach. Most production caching failures trace to prompt architecture decisions that teams never think of as caching decisions, and each one is fixable without changing providers.

Copy link to headingMove request-specific content out of the cached prefix

A timestamp, session ID, per-user name, or freshly retrieved document placed anywhere before the cache boundary invalidates the entire prefix behind it, not only that field. It is the single most common cause of a zero hit rate on a workload that looks correctly configured.

The fix is ordering. Put tool definitions and static instructions first, keep background documents behind that stable material, and move everything request-specific to the tail, after the cache boundary. In one DigitalOcean benchmark on Claude Haiku 4.5 with a roughly 6,000-token prefix, moving a single dynamic field out of the cached block took the hit rate from 0% to 99.3% and cut input cost from $6.72 to $0.57 per 1,000 requests, a 90% reduction. ProjectDiscovery reported the same pattern on its security agent, where relocating dynamic working memory to the end of the prompt raised the hit rate from 7% to 74% in a single deployment and cut inference cost 59%.

Copy link to headingFreeze tool-definition serialization order

Anthropic builds cache prefixes in a fixed order, tools then system then messages, which puts tool definitions at the top of the cached region. Runtime tool discovery over Model Context Protocol (MCP) servers, or schema assembly with unstable JSON key order, produces structurally identical tools that serialize to different bytes and miss the cache on every call. Changes to tool_choice invalidate cached message blocks as well.

Sort tool definitions deterministically and freeze the serialization order before sending. Anthropic now supports adding or removing tools mid-conversation without busting the cached prefix, which removes one common reason agentic runs lose their cache between steps.

Copy link to headingMatch the TTL tier to your read volume

A cache write with no follow-up read is a small net loss, because you pay the premium and never collect the discount. One-shot requests are the clearest case, paying 1.25× to 2× to write an entry that nothing reads back. For strictly one-shot traffic, manual markers or no caching beat caching: 'auto'.

Agentic workloads pull the other way. They combine tool lists assembled per request with session context in the system prompt, and inter-step gaps can outlive the 5-minute TTL. For bursty batch work that runs many requests in an hour and then goes quiet, the 1-hour tier wins even at 2× write cost, because the write is paid once and reads accumulate across the burst.

Copy link to headingInstrument cache reads and writes

Cache hit rate belongs on the same dashboard as latency and cost, because a drop often precedes a bill spike. Most provider APIs return cache usage in response metadata, so the signal is there to log and alert on.

When cache_creation_input_tokens and cache_read_input_tokens are both zero on requests that should be caching, audit the prompt architecture before touching any provider parameter. Two zeros almost always point to a request-specific field in the prefix or a prompt under the model's minimum, not to a misconfigured provider.

Copy link to headingHow Vercel AI SDK and AI Gateway unify prompt caching

Every provider exposes caching differently, which turns a multi-provider app into a stack of special cases, with Anthropic markers in one path, OpenAI keys in another, and implicit behavior somewhere else. The configuration surface is widest exactly where caching pays off most. Tool-call requests now carry 58.9% of tokens through AI Gateway, up from 31.6% in October 2025, and agentic traffic is both the biggest caching opportunity and the easiest place to break a prefix. AI SDK and AI Gateway expose caching controls at both the SDK and gateway layers so teams manage those differences in one place.

Copy link to headingOne providerOptions surface across every provider

Teams building across providers otherwise write and maintain a caching path per vendor. The AI SDK collapses that into one providerOptions surface. For Anthropic, set cacheControl: { type: 'ephemeral' } on a system message or content block, and { type: 'ephemeral', ttl: '1h' } to opt into the 1-hour TTL:

import { anthropic } from '@ai-sdk/anthropic';
import { generateText } from 'ai';

const result = await generateText({
  model: anthropic('claude-sonnet-5'),
  messages: [
    {
      role: 'system',
      content: BIG_SYSTEM_PROMPT,
      providerOptions: {
        anthropic: { cacheControl: { type: 'ephemeral', ttl: '1h' } },
      },
    },
    { role: 'user', content: userQuestion },
  ],
});

// Cache token counts come back on the usage object:
console.log(result.usage.inputTokenDetails.cacheReadTokens);
console.log(result.usage.inputTokenDetails.cacheWriteTokens);

AI SDK added tool-level caching through providerOptions on tool definitions, and cache usage comes back on the standard usage object for both generateText and streamText. In chat apps, the UIMessage type used by useChat does not carry providerOptions, so chat history has to pass through convertToModelMessages before markers attach.

Copy link to headingcaching: 'auto' handles explicit-marker providers for you

Some providers require markers and some do not, and keeping track of which is which is its own maintenance cost. At the gateway layer, providerOptions: { gateway: { caching: 'auto' } } adds cache markers automatically for the providers that need them, Anthropic and MiniMax, while OpenAI, Google, and DeepSeek cache without markers and pass through untouched:

const result = await generateText({
  model: 'anthropic/claude-sonnet-5',
  providerOptions: {
    gateway: { caching: 'auto' },
  },
  messages: [/* system prompt, then user turn */],
});

Automatic caching has a TTL limitation. caching: 'auto' applies only the default 5-minute cache, with no option to set Anthropic's 1-hour TTL. Workloads that depend on the 1-hour tier should set cache markers manually with ttl: '1h' or call the provider directly, since a gateway can silently downgrade a cache entry and a 90% cached-token discount changes your unit economics.

Copy link to headingSee cache reads and writes per request

Multi-provider caching is only manageable if you can see it per request. AI Gateway surfaces cache reads and writes without a custom telemetry layer. The Custom Reporting API returns cached_input_tokens and cache_creation_input_tokens per row, the Logs page breaks out reads and writes per request, and cachedInputTokens is available in onStepFinish callbacks for step-by-step runs.

That visibility is what turns a suspected caching regression into a specific fix. When the per-request breakdown shows zero reads and zero writes on traffic that should cache, the audit starts at the prompt, not the provider.

Copy link to headingRoute to models that cache automatically

Sometimes the reliable win is a model that caches without markers at all. The has: ['implicit-caching'] capability filter restricts routing to models that cache automatically, which helps when cache hit probability matters more than the specific model. Paired with cross-provider failover, it also keeps caching working through a provider incident.

Failover is not a side benefit here. Across the AI Gateway fleet, 3.5% of requests complete only after failing over to a second provider, and those rescued calls carry 4.9% of spend because long-context and multi-step runs hit limits first. Routing that preserves caching while surviving an outage is the difference between a degraded response and a user-visible error.

Copy link to headingShip cache-efficient AI apps on Vercel

A zero cache hit rate almost never means you picked the wrong provider. It means a request-specific field sits in the cached prefix, or the prompt falls under the model's minimum, or a gateway quietly reset the entry in flight. The teams that keep hit rates high treat prompt structure as the primary lever and the provider parameter as the last one. The platform underneath decides how much of that they manage by hand.

Here is how Vercel collapses cross-provider caching onto a small set of primitives teams can adopt without building a per-vendor caching layer:

• AI Gateway caching: 'auto': Adds the cache markers explicit-marker providers require, so Anthropic and MiniMax cache through the same code path OpenAI, Google, and DeepSeek use automatically.

• One providerOptions surface: Sets cache control, TTL, and tool-level caching across every provider through the AI SDK, so switching models does not mean rewriting the caching path.

• Per-request cache observability: Reports cached_input_tokens and cache_creation_input_tokens through the Custom Reporting API and the Logs page, so a hit-rate regression surfaces before it reaches an invoice.

• Implicit-caching capability filter: Routes to models that cache automatically with has: ['implicit-caching'] when hit probability matters more than model choice.

• Cross-provider failover: Keeps caching working through a provider incident by routing to a second provider, so an outage degrades gracefully instead of failing the request.

Start a new project to put these patterns into production, or browse Vercel templates for AI apps already wired for multi-provider routing.

Copy link to headingFAQs about prompt caching

Copy link to headingDoes routing through AI Gateway reset Anthropic prompt cache hits?

No. Provider-side prompt caching passes through by default, and caching: 'auto' adds Anthropic markers for you. Anything that mutates the prefix in flight can still invalidate a hit, so when the cached-token discount drives your unit economics, call the provider directly for full control.

Copy link to headingWhy is my cache hit rate zero even though I set cacheControl to ephemeral?

Usually a request-specific field sits before the cache boundary. Check the system prompt for timestamps, user IDs, session context, or retrieved documents. Also confirm the prefix clears the model's minimum, since Claude Haiku 4.5 needs 4,096 tokens and shorter prompts cache nothing with no error.

Copy link to headingDoes OpenAI still cache for free?

On models before GPT-5.6, caching is automatic with no write fee. From GPT-5.6 on, cache writes bill at 1.25× the uncached input rate and appear separately in the usage response, while reads keep a discount of up to 90%.

Copy link to headingWhat is the difference between prompt caching and semantic caching?

Prompt caching reuses KV tensors for a shared prefix inside the inference engine, and the model still generates a fresh response. Semantic caching returns a stored response when a new prompt is similar enough by embedding distance, so the model does not run and correctness depends on the similarity threshold.

Copy link to headingHow do I monitor cache hit rates through AI Gateway?

Read cachedInputTokens in onStepFinish callbacks, pull cached_input_tokens and cache_creation_input_tokens per row from the Custom Reporting API, and inspect per-request breakdowns on the Logs page. Two zeros on requests that should cache point to prompt structure first, not gateway configuration.