LLM cost management: how to track and control model spend in production

A billing alert fires and nobody can say whether the spike came from one user's agent loop or a continuous integration (CI) pipeline left running over a weekend. The alert did its job. What's missing is attribution, the request log that ties spend to a feature, a user, or a bad deploy. Spend alerts tell you something is wrong. Attribution tells you where and why. Controlling LLM cost in production starts with attribution, moves to enforcing per-key budgets, and ends with routing and caching that shrink the token surface before any cap has to trigger.

Copy link to headingKey takeaways

• Spend alerts show that something is wrong. Per-feature attribution shows where and why, so a spike traces to a responsible feature and model in minutes.

• Cheaper per-token prices do not lower bills on their own. On the AI Gateway network, spend grew 43% month over month in May 2026 while per-token prices held, because workloads shifted toward heavier, higher-quality models.

• Per-key budgets enforce rather than alert. When a key exceeds its cap, AI Gateway returns HTTP 402 and rejects further requests, so a runaway agent loop becomes a rejected request instead of a surprise invoice.

• Routing rescues spend that a direct provider path loses. Across the fleet, 3.5% of requests and 5.1% of tokens complete only after failing over to another provider.

• Prompt caching and cost-sorted routing shrink the token surface before any cap needs to trigger.

• Agentic loops need explicit step limits, because a per-agent cap misses loops that compound when one agent calls another.

Copy link to headingWhat is LLM cost management?

LLM cost management is the practice of attributing, capping, and reducing the money an application spends on model inference in production. It combines per-feature and per-user attribution, enforced spend limits, and token-reduction techniques like routing and caching into one operating discipline, rather than a monthly reconciliation against an invoice.

The discipline exists because inference cost behaves differently from traditional compute cost. A request's price depends on token count, model choice, and how many times an agent loops, none of which show up in a CPU or memory graph. Two teams running identical traffic can pay bills that differ widely based on which models they route to and how much context each request carries.

Copy link to headingWhy cheaper tokens still mean bigger LLM bills

Finding the cheapest model per task looks like the obvious first move. In practice the bill often grows anyway. The June production index shows total AI Gateway spend up 43% month over month in May 2026 while token volume grew 20%, with teams paying almost 20% more per token on average than the month before. Provider prices held steady while the workload shifted toward higher-quality, higher-cost models.

The mechanism is Jevons paradox. Cheaper intelligence pushes teams to run more agents and automate more workflows, and code generation adds still more usage, so aggregate spend rises even as the unit cost of a token falls.

Agents amplify the effect. On the AI Gateway network, tool-call requests grew from 31.6% to 58.9% of all tokens between October 2025 and April 2026, and tool-using requests run about 2.6 times more token-heavy than the rest, per the production index. An agent that fires ten tool calls bills roughly ten times the tokens a single chat turn would. Tracking aggregate consumption matters more than shaving the per-token price, because unit costs can fall while the total climbs.

Copy link to headingWhat you need before you track LLM spend

Attribution and budgets assume a few things are already in place. Set these up before touching a spend dashboard.

• A Vercel account on the right plan: The Custom Reporting API runs on the paid tier. Every team gets $5 per month in AI Gateway credits with free-tier models, and lower rate limits apply until you add credits.

• AI Gateway at the team level: Bring Your Own Key (BYOK) requires the paid tier, and it lets you attribute spend across both your own provider keys and system credentials.

• A current AI SDK or a compatible client: The examples here use AI SDK 7, which routes through AI Gateway by default. The gateway also accepts OpenAI Chat Completions and Anthropic Messages calls through one endpoint.

• A spend baseline: Set caps against at least one week of request logs. With no history, start by instrumenting and watching before you enforce.

• A budget for observability itself: Custom Reporting bills $0.075 per 1,000 writes and $5 per 1,000 queries at current pricing, deducted from your AI Gateway credits.

These prerequisites gate everything downstream, since a cap set without a baseline either throttles real traffic or never fires.

Copy link to headingAttribute model spend by feature and user

Provider dashboards report aggregates, not causes. The OpenAI dashboard groups by project, and a per-key cost breakdown means creating a separate project per key. A $12,000 monthly jump could be one customer's agent loop or a retry storm during an upstream outage, and the aggregate view can't tell them apart.

Attribution starts with tagging every request. Spend is one signal in a broader observability stack, and the gateway captures it with no instrumentation because it sits in the request path. Attach providerOptions.gateway.user and providerOptions.gateway.tags on each call, so both propagate to the observability dashboard and the Custom Reporting API:

import { generateText } from 'ai';

const result = await generateText({
  model: 'anthropic/claude-sonnet-4.6',
  prompt,
  providerOptions: {
    gateway: {
      user: 'user_1234567890',
      tags: ['checkout', 'production'],
    },
  },
});

Every response also carries a generation ID, surfaced through providerMetadata.gateway.generationId, so a single expensive call traces back from a report to its exact request.

With tags flowing, turn on Custom Reporting. It returns per-day cost and request volume grouped by model, provider, user ID, or tag, across both BYOK and system credentials, and its token counts include cached, cache-creation, and reasoning tokens. The first query worth running sorts by total_cost grouped by user and tag. Any user or feature over 20% of spend earns a dedicated cap before anything fleet-wide. Consolidating attribution into one system is what let one AI platform serving more than 200,000 users retire a separate proxy layer and save over $80,000.

Copy link to headingCap LLM spend with per-key budgets

Attribution tells you where the money goes. A budget stops it from going too far. Cost alerts typically fire after the threshold is crossed, so the money is already spent. A per-key budget rejects the request instead. Set the cap at key creation or in the dashboard, and when the key exceeds it, AI Gateway returns HTTP 402 and rejects further requests on that key until the budget resets or you raise it. One cap covers every provider and model on the key.

Create a scoped key from the CLI:

vercel ai-gateway api-keys create --name checkout-prod --budget 10 --refresh-period monthly

Supported refresh periods are daily, weekly, monthly, or none, with a $1 minimum budget. Issue a separate key per environment and per major feature, because a single team-wide key creates one shared cap that starves unrelated features when any one of them misbehaves. Scoped keys bound the blast radius to the feature that caused the overrun.

Team-wide controls complement per-key budgets rather than replace them. On Pro, Spend Management fires webhooks at 50% and 75% of the configured amount, then again at 100%, with checks running every few minutes, so a pause can lag the actual crossing. Set the limit below your true ceiling to absorb that lag. For workloads that need continuity, auto-recharge tops up credits before the balance hits zero. It's off by default, and it belongs behind its own monthly recharge limit so a runaway workload can't refill itself indefinitely.

Copy link to headingCut LLM costs with routing and caching

Budgets defend a ceiling. Routing and caching lower the floor, so the ceiling gets hit less often. Across the AI Gateway fleet, 3.5% of requests and 5.1% of tokens complete only after failing over to another provider, per the production index. On a direct-to-provider path, those requests return errors. Routing cuts cost and rescues requests at the same time.

Most of these techniques are native to AI Gateway or the AI SDK. Read the table for which layer each one lives in, its measured effect, and when to reach for it:

Turn on cost-sorted routing and prompt caching before tightening any budget. The lowest-cost provider can be slower, so when latency matters more than price, sort by ttft instead. Caching rewards prefix discipline. Put static content first, because a timestamp inside the cached prefix produces zero hits and a cache-write charge on every call. For a deeper setup, the cost-aware routing guide walks through the configuration end to end.

Copy link to headingManage agentic loops as a separate LLM cost class

Routing and caching handle the steady state. Agent loops break the steady-state assumption, because they re-send the full accumulated context on every step, so cost grows with the square of the loop length rather than linearly. A ten-step agent carrying a 4,000-token system prompt and 500-token average tool outputs has pushed past 40,000 input tokens by its final turn, most of it re-reading context it already paid for.

The tools an agent can reach move cost more than model choice does. When we removed 80% of an internal agent's tools in a published benchmark, execution time dropped from 274.8 seconds to 77.4 seconds, the success rate rose from 80% to 100%, and token usage fell 37%, from roughly 102,000 to 61,000 tokens per run. That agent ran Claude Opus 4.5 through the AI SDK, so exposing fewer tools shifted cost and reliability more than any routing change would have.

Cap the loop explicitly. The AI SDK stops a tool-calling loop after a set number of steps with stopWhen:

import { generateText, stepCountIs } from 'ai';

const result = await generateText({
  model: 'anthropic/claude-sonnet-4.6',
  tools,
  stopWhen: stepCountIs(10),
});

Framework-level max_iterations settings miss loops that compound across agent boundaries, since a per-agent cap can't see the steps a sub-agent takes when it's called through a tool. Give agent-facing features their own key with a lower daily budget, so a runaway loop hits its own cap before it drains the team balance. Watch reasoning models separately as well, because some providers bill thinking tokens as output and may summarize rather than return them in full. Custom Reporting exposes reasoning_tokens where providers report them.

Copy link to headingThree failure modes that break LLM spend controls in production

Cost controls fail in production for reasons that have nothing to do with the model. Three patterns account for most of it.

1. Surfacing 402 and 503 errors raw to users: When a key's budget is exhausted, the 402 needs application-layer handling. Catch it and route to a cheaper model, a feature-off state, or a cached response. The models fallback array switches models when the primary fails and same-model provider failover is automatic, but budget exhaustion is a policy decision your code has to make.

2. Dev and CI pipelines sharing a production key: Every CI run against live model APIs burns real tokens at production rates, and a flaky test's retry loop can drain a shared daily budget before the first user request of the morning. Give development and CI their own keys with tighter daily caps.

3. Applying a fleet-wide cap before per-feature attribution exists: The Spend Management hard cap pauses every project when it fires, and paused projects return 503 DEPLOYMENT_PAUSED to users. Without attribution, one heavy feature or one bad actor takes down everything equally. Scope per-key budgets to features first, and keep the fleet-wide cap as a last resort.

Each of these is cheaper to prevent than to diagnose after an incident, and each maps to a key-scoping decision you can make before the first production request.

Copy link to headingHow Vercel AI Gateway powers LLM cost management

The tactics above assume a platform that can attribute, enforce, and route without a team building each piece itself. An AI gateway provides those as managed capabilities, and AI Gateway is Vercel's implementation, so LLM cost management becomes configuration rather than infrastructure work.

Copy link to headingAttribution across BYOK and system credentials

Teams that bring their own provider keys usually lose a unified cost view, because BYOK spend and platform spend land in different places. The Custom Reporting API reports both in one query, grouped by model, provider, user, or tag, so margin analysis and per-customer usage come from a single source instead of a reconciliation spreadsheet.

Copy link to headingProvider arbitrage on the same model

The same model often lists at different prices across providers, and a direct integration locks you to one of them. Cost-sorted routing (sort: 'cost') sends each request to the cheapest provider that serves the model, and reprices as availability shifts, so arbitrage happens per request without touching application code. Pair it with an explicit models fallback chain so price-shopping never costs you a request when a provider degrades.

Copy link to headingZero-markup routing with built-in failover

Direct provider integrations trade away failover, and some gateways add a per-token markup on top. AI Gateway bills at provider list price with no markup, including on BYOK requests, and reissues a failed request to a healthy provider automatically. That failover is why 3.5% of fleet requests still succeed instead of returning errors a team would otherwise pay to retry.

Copy link to headingOne endpoint and one meter for every provider

Running OpenAI in one service and Anthropic in another leaves a team reconciling separate keys, dashboards, and bills. AI Gateway reaches hundreds of models through a single endpoint with one meter, so spend, latency, and token counts report in one place regardless of how many providers sit behind them.

Copy link to headingShip a predictable, attributable LLM bill on Vercel

The team debugging an invoice at the start of this guide had the alert but not the attribution. Cost controls work when that gap closes, when a spike traces within minutes to the feature and model responsible, and the cap that catches it is scoped to that surface rather than the whole fleet. Getting there means attributing first, enforcing with per-key budgets, then shrinking the token surface so the budgets rarely bind.

Here is how Vercel turns that sequence into managed infrastructure:

• Per-key budgets with 402 enforcement: A key that exceeds its cap rejects requests instead of alerting after the spend, so a runaway loop becomes a rejected call rather than a surprise invoice.

• Custom Reporting by user, tag, and credential type: Per-day cost and token counts, including cached and reasoning tokens, across BYOK and system credentials in one query.

• Zero-markup BYOK pricing: Tokens bill at the provider's list price, including when you bring your own key.

• Prompt caching passthrough: Provider-native prefix caching passes through automatically, cutting repeat-prefix token costs before a request bills.

• Cost-sorted routing with automatic failover: Requests route to the cheapest qualifying provider and fail over to a healthy one without downtime.

Start a new project at vercel.com/new and route your first model call through AI Gateway, then add per-key budgets and tags as traffic grows. Browse vercel.com/templates for AI apps already wired for spend control.

Copy link to headingFrequently asked questions about LLM cost management

Copy link to headingDoes Vercel AI Gateway add markup on top of provider pricing?

No. AI Gateway bills at provider list prices with no markup on inference, including on BYOK requests. Some optional capabilities that are off by default add charges when enabled, but the per-token cost itself matches the provider's list rate.

Copy link to headingWhat happens if an API key budget is exhausted mid-request?

AI Gateway returns HTTP 402 and rejects the request. The key stays blocked until the budget resets on its configured refresh period, daily, weekly, or monthly, or until you raise the limit in the dashboard. Application code should catch the 402 and degrade gracefully.

Copy link to headingDoes AI Gateway support semantic caching to cut repeated calls?

No. Semantic caching is out of AI Gateway's scope. Provider-native prompt caching passes through automatically and is the first caching layer to enable. Semantic caching needs an application-layer implementation with a vector store to match requests by meaning rather than exact prefix.

Copy link to headingCan I track BYOK and system-credential costs in the same report?

Yes. The Custom Reporting API covers both BYOK and system credentials in one place, with a credential_type grouping when you need the split. That single view is what lets teams calculate margins across providers without stitching separate billing exports together.

Copy link to headingHow do I stop one user's agent from draining the team budget?

Issue a separate API key per user tier or feature, each with its own daily budget. When that key hits its limit, the 402 isolates the impact to it while every other key keeps serving. For multi-tenant platforms, this maps to per-tenant keys with per-tenant spend caps.