AI Gateway Provider Latency and Throughput Metrics

AI Gateway tracks throughput and latency for each model and provider, measured from live AI Gateway traffic. For provider availability, see uptime.

Throughput is the rate at which a provider returns output tokens, measured in tokens per second. Higher throughput means a provider returns output tokens faster.

Latency is time to first token (TTFT), measured in milliseconds. It's the duration between the request reaching the provider and the first response token arriving. Lower latency means a faster perceived response.

You can find metrics in the following places:

• The AI Gateway model catalog: each model row shows the best metrics across all providers (highest throughput, lowest latency).
• The AI Gateway > Models tab in your project: same view as the catalog, scoped to your team.

To see metrics broken down by provider, click any model to open its detail page. For each provider, the detail page shows the live values from the last hour and a chart of how each metric has trended over time.

Select Share in the top-right corner to export the current view as an image. The export captures the selected model, metric, and time range so you can share a snapshot in tickets, postmortems, or status updates.

Each endpoint returned by the model endpoints API includes rolling one-hour metrics for that provider:

curl -i https://ai-gateway.vercel.sh/v1/models/anthropic/claude-opus-5/endpoints \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY"

Replace anthropic/claude-opus-5 with the model ID you want to inspect, and set AI_GATEWAY_API_KEY to a valid AI Gateway API key. Each endpoint object includes:

Example response excerpt:

{
  "data": {
    "id": "anthropic/claude-opus-5",
    "endpoints": [
      {
        "provider_name": "anthropic",
        "throughput_last_1h": { "p50": 67, "p95": 69.85 },
        "latency_last_1h": { "p50": 2292, "p95": 2685 }
      }
    ]
  }
}

Throughput and latency are measured from requests that run directly through AI Gateway. Bring Your Own Key (BYOK) requests aren't included since their performance depends on your own provider credentials, not AI Gateway's.

These numbers reflect what AI Gateway observes when serving live traffic. They may differ from what providers report directly due to factors specific to AI Gateway, like account tiering, regions, or routing decisions.