Advanced

OpenAI Chat Completions Configuration with AI Gateway

Control which providers serve your request, what happens when one fails, and how much of your prompt is cached between calls. For controlling how much a model thinks before answering, see Reasoning.

Provider options

The AI Gateway can route your requests across multiple AI providers for better reliability and performance. You can control which providers are used and in what order through the providerOptions parameter.

Example request

provider-options.ts

import OpenAI from 'openai';
 
const apiKey = process.env.AI_GATEWAY_API_KEY || process.env.VERCEL_OIDC_TOKEN;
 
const openai = new OpenAI({
  apiKey,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const completion = await openai.chat.completions.create({
  model: 'anthropic/claude-opus-5',
  messages: [
    {
      role: 'user',
      content:
        'Tell me the history of the San Francisco Mission-style burrito in two paragraphs.',
    },
  ],
  stream: false,
  // Provider options for gateway routing preferences
  // AI Gateway extension fields are not included in the upstream SDK types.
  ...{
    providerOptions: {
      gateway: {
        order: ['vertex', 'anthropic'], // Try Vertex AI first, then Anthropic
      },
    },
  },
});
 
console.log('Assistant:', completion.choices[0].message.content);
console.log('Tokens used:', completion.usage);

provider-options.py

import os
from openai import OpenAI
 
api_key = os.getenv('AI_GATEWAY_API_KEY') or os.getenv('VERCEL_OIDC_TOKEN')
 
client = OpenAI(
    api_key=api_key,
    base_url='https://ai-gateway.vercel.sh/v1'
)
 
completion = client.chat.completions.create(
    model='anthropic/claude-opus-5',
    messages=[
        {
            'role': 'user',
            'content': 'Tell me the history of the San Francisco Mission-style burrito in two paragraphs.'
        }
    ],
    stream=False,
    # Provider options for gateway routing preferences
    extra_body={
        'providerOptions': {
            'gateway': {
                'order': ['vertex', 'anthropic']  # Try Vertex AI first, then Anthropic
            }
        }
    }
)
 
print('Assistant:', completion.choices[0].message.content)
print('Tokens used:', completion.usage)

provider-options.sh

curl -X POST "https://ai-gateway.vercel.sh/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-opus-5",
    "messages": [
      {
        "role": "user",
        "content": "Tell me the history of the San Francisco Mission-style burrito in two paragraphs."
      }
    ],
    "stream": false,
    "providerOptions": {
      "gateway": {
        "order": [
          "vertex",
          "anthropic"
        ]
      }
    }
  }'

Provider routing: In this example, the gateway will first attempt to use Vertex AI to serve the Claude model. If Vertex AI is unavailable or fails, it will fall back to Anthropic. Other providers are still available but will only be used after the specified providers.

Model fallbacks

You can specify fallback models that will be tried in order if the primary model fails. There are two ways to do this:

Option 1: Direct `models` field

The simplest way is to use the models field directly at the top level of your request:

model-fallbacks.ts

import OpenAI from 'openai';
 
const apiKey = process.env.AI_GATEWAY_API_KEY || process.env.VERCEL_OIDC_TOKEN;
 
const openai = new OpenAI({
  apiKey,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const completion = await openai.chat.completions.create({
  model: 'openai/gpt-6-astra', // Primary model
  // @ts-ignore - models is a gateway extension
  ...{
    models: ['anthropic/claude-opus-5', 'google/gemini-3.6-flash'],
  }, // Fallback models
  messages: [
    {
      role: 'user',
      content: 'Write a haiku about TypeScript.',
    },
  ],
  stream: false,
});
 
console.log('Assistant:', completion.choices[0].message.content);
 
// Check which model was actually used
console.log('Model used:', completion.model);

model-fallbacks.py

import os
from openai import OpenAI
 
api_key = os.getenv('AI_GATEWAY_API_KEY') or os.getenv('VERCEL_OIDC_TOKEN')
 
client = OpenAI(
    api_key=api_key,
    base_url='https://ai-gateway.vercel.sh/v1'
)
 
completion = client.chat.completions.create(
    model='openai/gpt-6-astra',  # Primary model
    messages=[
        {
            'role': 'user',
            'content': 'Write a haiku about TypeScript.'
        }
    ],
    stream=False,
    # models is a gateway extension for fallback models
    extra_body={
        'models': ['anthropic/claude-opus-5', 'google/gemini-3.6-flash']  # Fallback models
    }
)
 
print('Assistant:', completion.choices[0].message.content)
 
# Check which model was actually used
print('Model used:', completion.model)

model-fallbacks.sh

curl -X POST "https://ai-gateway.vercel.sh/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-6-astra",
    "models": [
      "anthropic/claude-opus-5",
      "google/gemini-3.6-flash"
    ],
    "messages": [
      {
        "role": "user",
        "content": "Write a haiku about TypeScript."
      }
    ],
    "stream": false
  }'

Option 2: Via provider options

Alternatively, you can specify model fallbacks through the providerOptions.gateway.models field:

model-fallbacks-provider-options.ts

import OpenAI from 'openai';
 
const apiKey = process.env.AI_GATEWAY_API_KEY || process.env.VERCEL_OIDC_TOKEN;
 
const openai = new OpenAI({
  apiKey,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const completion = await openai.chat.completions.create({
  model: 'openai/gpt-6-astra', // Primary model
  messages: [
    {
      role: 'user',
      content: 'Write a haiku about TypeScript.',
    },
  ],
  stream: false,
  // Model fallbacks via provider options
  ...{
    providerOptions: {
      gateway: {
        models: ['anthropic/claude-opus-5', 'google/gemini-3.6-flash'], // Fallback models
      },
    },
  },
});
 
console.log('Assistant:', completion.choices[0].message.content);
console.log('Model used:', completion.model);

model-fallbacks-provider-options.py

import os
from openai import OpenAI
 
api_key = os.getenv('AI_GATEWAY_API_KEY') or os.getenv('VERCEL_OIDC_TOKEN')
 
client = OpenAI(
    api_key=api_key,
    base_url='https://ai-gateway.vercel.sh/v1'
)
 
completion = client.chat.completions.create(
    model='openai/gpt-6-astra',  # Primary model
    messages=[
        {
            'role': 'user',
            'content': 'Write a haiku about TypeScript.'
        }
    ],
    stream=False,
    # Model fallbacks via provider options
    extra_body={
        'providerOptions': {
            'gateway': {
                'models': ['anthropic/claude-opus-5', 'google/gemini-3.6-flash']  # Fallback models
            }
        }
    }
)
 
print('Assistant:', completion.choices[0].message.content)
print('Model used:', completion.model)

model-fallbacks-provider-options.sh

curl -X POST "https://ai-gateway.vercel.sh/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-6-astra",
    "messages": [
      {
        "role": "user",
        "content": "Write a haiku about TypeScript."
      }
    ],
    "stream": false,
    "providerOptions": {
      "gateway": {
        "models": [
          "anthropic/claude-opus-5",
          "google/gemini-3.6-flash"
        ]
      }
    }
  }'

Which approach to use: Both methods achieve the same result. Use the direct models field (Option 1) for simplicity, or use providerOptions (Option 2) if you're already using provider options for other configurations.

Both configurations will:

Try the primary model (openai/gpt-6-astra) first
If it fails, try anthropic/claude-opus-5
If that also fails, try google/gemini-3.6-flash
Return the result from the first model that succeeds

Streaming with provider options

Provider options work with streaming requests as well:

streaming-provider-options.ts

import OpenAI from 'openai';
 
const apiKey = process.env.AI_GATEWAY_API_KEY || process.env.VERCEL_OIDC_TOKEN;
 
const openai = new OpenAI({
  apiKey,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const stream = await openai.chat.completions.create({
  model: 'anthropic/claude-opus-5',
  messages: [
    {
      role: 'user',
      content:
        'Tell me the history of the San Francisco Mission-style burrito in two paragraphs.',
    },
  ],
  stream: true,
  ...{
    providerOptions: {
      gateway: {
        order: ['vertex', 'anthropic'],
      },
    },
  },
});
 
for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content;
  if (content) {
    process.stdout.write(content);
  }
}

streaming-provider-options.py

import os
from openai import OpenAI
 
api_key = os.getenv('AI_GATEWAY_API_KEY') or os.getenv('VERCEL_OIDC_TOKEN')
 
client = OpenAI(
    api_key=api_key,
    base_url='https://ai-gateway.vercel.sh/v1'
)
 
stream = client.chat.completions.create(
    model='anthropic/claude-opus-5',
    messages=[
        {
            'role': 'user',
            'content': 'Tell me the history of the San Francisco Mission-style burrito in two paragraphs.'
        }
    ],
    stream=True,
    extra_body={
        'providerOptions': {
            'gateway': {
                'order': ['vertex', 'anthropic']
            }
        }
    }
)
 
for chunk in stream:
    content = chunk.choices[0].delta.content
    if content:
        print(content, end='', flush=True)

streaming-provider-options.sh

curl -X POST "https://ai-gateway.vercel.sh/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-opus-5",
    "messages": [
      {
        "role": "user",
        "content": "Tell me the history of the San Francisco Mission-style burrito in two paragraphs."
      }
    ],
    "stream": true,
    "providerOptions": {
      "gateway": {
        "order": [
          "vertex",
          "anthropic"
        ]
      }
    }
  }'

For more details about available providers and advanced provider configuration, see the Provider Options documentation.

Provider sorting

Use the sort option to rank providers by cost, latency, or throughput. The gateway sorts the available providers by the chosen metric and tries them in that order.

Value	Description	Direction
`'cost'`	Sort by estimated cost	Lowest cost first
`'ttft'`	Sort by time to first token (median, in ms)	Lowest latency first
`'tps'`	Sort by tokens per second throughput (median)	Highest first

You can pass sort through providerOptions.gateway:

sort-provider-options.ts

import OpenAI from 'openai';
 
const apiKey = process.env.AI_GATEWAY_API_KEY || process.env.VERCEL_OIDC_TOKEN;
 
const openai = new OpenAI({
  apiKey,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const completion = await openai.chat.completions.create({
  model: 'anthropic/claude-sonnet-5',
  messages: [
    {
      role: 'user',
      content: 'What is 2 + 2? Answer in one sentence.',
    },
  ],
  stream: false,
  ...{
    providerOptions: {
      gateway: {
        sort: 'tps', // Use the highest throughput provider first
      },
    },
  },
});
 
console.log('Assistant:', completion.choices[0].message.content);

sort-provider-options.py

import os
from openai import OpenAI
 
api_key = os.getenv('AI_GATEWAY_API_KEY') or os.getenv('VERCEL_OIDC_TOKEN')
 
client = OpenAI(
    api_key=api_key,
    base_url='https://ai-gateway.vercel.sh/v1'
)
 
completion = client.chat.completions.create(
    model='anthropic/claude-sonnet-5',
    messages=[
        {
            'role': 'user',
            'content': 'What is 2 + 2? Answer in one sentence.'
        }
    ],
    stream=False,
    extra_body={
        'providerOptions': {
            'gateway': {
                'sort': 'tps'  # Use the highest throughput provider first
            }
        }
    }
)
 
print('Assistant:', completion.choices[0].message.content)

sort-provider-options.sh

curl -X POST "https://ai-gateway.vercel.sh/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [
      {
        "role": "user",
        "content": "What is 2 + 2? Answer in one sentence."
      }
    ],
    "stream": false,
    "providerOptions": {
      "gateway": {
        "sort": "tps"
      }
    }
  }'

Or use the provider shorthand directly in the request body:

sort-provider-shorthand.ts

import OpenAI from 'openai';
 
const apiKey = process.env.AI_GATEWAY_API_KEY || process.env.VERCEL_OIDC_TOKEN;
 
const openai = new OpenAI({
  apiKey,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const completion = await openai.chat.completions.create({
  model: 'anthropic/claude-sonnet-5',
  messages: [
    {
      role: 'user',
      content: 'What is 2 + 2? Answer in one sentence.',
    },
  ],
  stream: false,
  ...{
    provider: {
      sort: 'tps',
    },
  },
});
 
console.log('Assistant:', completion.choices[0].message.content);

sort-provider-shorthand.py

import os
from openai import OpenAI
 
api_key = os.getenv('AI_GATEWAY_API_KEY') or os.getenv('VERCEL_OIDC_TOKEN')
 
client = OpenAI(
    api_key=api_key,
    base_url='https://ai-gateway.vercel.sh/v1'
)
 
completion = client.chat.completions.create(
    model='anthropic/claude-sonnet-5',
    messages=[
        {
            'role': 'user',
            'content': 'What is 2 + 2? Answer in one sentence.'
        }
    ],
    stream=False,
    extra_body={
        'provider': {
            'sort': 'tps'  # Use the highest throughput provider first
        }
    }
)
 
print('Assistant:', completion.choices[0].message.content)

sort-provider-shorthand.sh

curl -X POST "https://ai-gateway.vercel.sh/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [
      {
        "role": "user",
        "content": "What is 2 + 2? Answer in one sentence."
      }
    ],
    "stream": false,
    "provider": {
      "sort": "tps"
    }
  }'

The provider shorthand is equivalent to setting the same fields in providerOptions.gateway. If both are provided for the same option, they must resolve to the same value or the request will fail. For the full details on sorting behavior, metrics, and health interactions, see Provider Filtering, Ordering & Sorting.

Provider timeouts

You can set per-provider timeouts for BYOK credentials to trigger fast failover when a provider is slow to respond. Pass providerTimeouts in providerOptions.gateway:

"providerOptions": {
  "gateway": {
    "providerTimeouts": {
      "byok": { "anthropic": 3000, "bedrock": 5000 }
    }
  }
}

For full details, limits, and response metadata, see Provider Timeouts.

Request-scoped BYOK (Bring Your Own Key)

You can pass your own provider credentials on a per-request basis using the byok option in providerOptions.gateway. This allows you to use your existing provider accounts and access private resources without configuring credentials in the gateway settings.

Example request

byok.ts

import OpenAI from 'openai';
 
const apiKey = process.env.AI_GATEWAY_API_KEY || process.env.VERCEL_OIDC_TOKEN;
 
const openai = new OpenAI({
  apiKey,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const completion = await openai.chat.completions.create({
  model: 'anthropic/claude-opus-5',
  messages: [
    {
      role: 'user',
      content: 'Hello, world!',
    },
  ],
  ...{
    providerOptions: {
      gateway: {
        byok: {
          anthropic: [
            {
              apiKey: process.env.ANTHROPIC_API_KEY,
            },
          ],
        },
      },
    },
  },
});
 
console.log(completion.choices[0].message.content);

byok.py

import os
from openai import OpenAI
 
api_key = os.getenv('AI_GATEWAY_API_KEY') or os.getenv('VERCEL_OIDC_TOKEN')
 
client = OpenAI(
    api_key=api_key,
    base_url='https://ai-gateway.vercel.sh/v1'
)
 
completion = client.chat.completions.create(
    model='anthropic/claude-opus-5',
    messages=[
        {
            'role': 'user',
            'content': 'Hello, world!'
        }
    ],
    extra_body={
        'providerOptions': {
            'gateway': {
                'byok': {
                    'anthropic': [{'apiKey': os.getenv('ANTHROPIC_API_KEY')}]
                }
            }
        }
    }
)
 
print(completion.choices[0].message.content)

byok.sh

curl -X POST "https://ai-gateway.vercel.sh/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-opus-5",
    "messages": [
      { "role": "user", "content": "Hello, world!" }
    ],
    "providerOptions": {
      "gateway": {
        "byok": {
          "anthropic": [{ "apiKey": "'"$ANTHROPIC_API_KEY"'" }]
        }
      }
    }
  }'

The byok option is a record where keys are provider slugs and values are arrays of credential objects. Each provider can have multiple credentials that are tried in order.

Credential structure by provider:

Anthropic: { apiKey: string }
OpenAI: { apiKey: string }
Google Vertex AI: { project: string, location: string, googleCredentials: { privateKey: string, clientEmail: string } }
Amazon Bedrock: { accessKeyId: string, secretAccessKey: string, region?: string }

For detailed credential parameters for each provider, see the AI SDK providers documentation.

Multiple credentials example:

providerOptions: {
  gateway: {
    byok: {
      // Multiple credentials for the same provider (tried in order)
      vertex: [
        { project: 'proj-1', location: 'us-east5', googleCredentials: { privateKey: '...', clientEmail: '...' } },
        { project: 'proj-2', location: 'us-east5', googleCredentials: { privateKey: '...', clientEmail: '...' } },
      ],
      // Multiple providers
      anthropic: [{ apiKey: 'sk-ant-...' }],
    },
  },
},

Credential precedence: When request-scoped BYOK credentials are provided, any cached BYOK credentials configured in the gateway settings are not considered. Requests may still fall back to system credentials if the provided credentials fail. For persistent BYOK configuration, see the BYOK documentation.

Prompt caching

Anthropic Claude models support prompt caching, which can significantly reduce costs and latency for repeated prompts. You can enable caching automatically or manually.

Automatic caching

Use caching: 'auto' in providerOptions to let AI Gateway automatically add cache markers for providers that require them (like Anthropic). For full details, supported providers, and examples, see Automatic Caching.

Manual caching

For fine-grained control, you can manually mark content with cache_control:

prompt-caching.ts

import OpenAI from 'openai';
 
const apiKey = process.env.AI_GATEWAY_API_KEY || process.env.VERCEL_OIDC_TOKEN;
 
const openai = new OpenAI({
  apiKey,
  baseURL: 'https://ai-gateway.vercel.sh/v1',
});
 
const response = await openai.chat.completions.create({
  model: 'anthropic/claude-opus-5',
  messages: [
    {
      role: 'user',
      content: 'Analyze this document and summarize the key points.',
      ...{
        cache_control: {
          type: 'ephemeral',
        },
      },
    },
  ],
});
 
console.log(response.choices[0].message.content);

prompt-caching.py

import os
from openai import OpenAI
 
api_key = os.getenv('AI_GATEWAY_API_KEY') or os.getenv('VERCEL_OIDC_TOKEN')
 
client = OpenAI(
    api_key=api_key,
    base_url='https://ai-gateway.vercel.sh/v1'
)
 
response = client.chat.completions.create(
    model='anthropic/claude-opus-5',
    messages=[
        {
            'role': 'user',
            'content': 'Analyze this document and summarize the key points.',
            'cache_control': {
                'type': 'ephemeral'
            }
        }
    ]
)
 
print(response.choices[0].message.content)

manual-caching.sh

curl -X POST "https://ai-gateway.vercel.sh/v1/chat/completions" \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-opus-5",
    "messages": [
      {
        "role": "user",
        "content": "Analyze this document and summarize the key points.",
        "cache_control": {
          "type": "ephemeral"
        }
      }
    ]
  }'

Cache control types: The ephemeral cache type stores content for the duration of the session. This is useful for large system prompts, documents, or context that you want to reuse across multiple requests. Prompt caching works with Anthropic models across all supported providers (Anthropic, Vertex AI, and Bedrock). For more details, see Anthropic's prompt caching documentation.

Last updated September 8, 2026

Cross-link map: OpenAI Chat Completions Configuration with AI Gateway (/docs/ai-gateway/sdks-and-apis/openai-chat-completions/advanced)

From the Vercel docs graph (built 2026-10-09T15:41:22.984Z), spanning vercel.com docs + KB, nextjs.org, ai-sdk.dev, and other Vercel documentation sites. Full graph as JSON: https://vercel.com/docs/graph.json

Semantically closest pages

AI Gateway Provider Timeouts — Configure AI Gateway timeouts for BYOK providers. Fail over to the next provider when a request takes too long to start
AI Gateway Provider Routing and Fallbacks — Configure provider routing, ordering, and fallback behavior in Vercel AI Gateway.
AI Gateway Provider Filtering, Ordering, and Sorting — Control AI Gateway provider routing with order, only, and sort. Set preferences, restrict providers, and rank them by co
OpenResponses Configuration with AI Gateway — Configure provider routing, fallbacks, and restrictions using the OpenResponses API through AI Gateway.
Bring Your Own Key \(BYOK\) to AI Gateway — Learn how to configure your own provider keys with AI Gateway.

Prerequisites

AI Gateway — Vercel MCP tools for ai gateway.

This page links to (6)

AI SDK Providers — Learn how to use AI SDK providers.
AI Gateway Automatic Prompt Caching — Enable automatic prompt caching and provider-side cache affinity with AI Gateway to reduce costs and latency.
AI Gateway Provider Filtering, Ordering, and Sorting — Control AI Gateway provider routing with order, only, and sort. Set preferences, restrict providers, and rank them by co
AI Gateway Provider Routing and Fallbacks — Configure provider routing, ordering, and fallback behavior in Vercel AI Gateway.
AI Gateway Provider Timeouts — Configure AI Gateway timeouts for BYOK providers. Fail over to the next provider when a request takes too long to start
OpenAI Chat Completions Reasoning with AI Gateway — Control how much a model thinks before answering with the OpenAI Chat Completions API through AI Gateway.

Pages that link here (5)

By site: vercel-docs (5)

AI Gateway Automatic Prompt Caching — Enable automatic prompt caching and provider-side cache affinity with AI Gateway to reduce costs and latency.
OpenAI Chat Completions API with AI Gateway — Use OpenAI SDKs with the AI Gateway Chat Completions API. Configure the base URL and authentication for chat, streaming,
OpenAI Chat Completions Requests with AI Gateway — Create chat completions using the Chat Completions API with support for streaming, image attachments, and PDF documents
OpenAI Chat Completions Reasoning with AI Gateway — Control how much a model thinks before answering with the OpenAI Chat Completions API through AI Gateway.
OpenAI Chat Completions Streaming with AI Gateway — Stream OpenAI Chat Completions responses token by token as they are generated through AI Gateway.

Images

Embeddings

Was this helpful?