Decision Fallbacks

Decision fallbacks rerun a successful decision with another model when an answer is uncertain. You opt in per request or virtual model by adding one conditional object to providerOptions.gateway.models. Requests without a conditional models entry keep their existing behavior.

The models array accepts two entry types with different triggers:

A string entry can name a native decision model or a language model, with or without a conditional object in the request. A language model answers through structured output.

The conditional object must be the first entry in models, and models can contain only one conditional object. You can add string execution-error fallbacks after it:

{
  "gateway": {
    "models": [
      {
        "model": "openai/gpt-6-astra",
        "when": {
          "question": "intent",
          "confidenceBelow": 0.6
        }
      },
      "anthropic/claude-sonnet-5"
    ]
  }
}

AI Gateway supports one conditional object and one successful-response fallback stage in a request. String entries after the object remain available if the conditional model fails during execution.

If the primary model fails before returning answers, AI Gateway tries the conditional object's model under the execution-error fallback behavior. It cannot check when without a successful primary result.

Use confidenceBelow with a Choice or Score question. Choice and Score confidence measures the concentration of the answer's probability distribution. It is not the probability of a Boolean value.

The following example escalates an uncertain Choice answer to a language model:

import { gateway, type GatewayProviderOptions } from '@ai-sdk/gateway';
import { experimental_decide as decide } from 'ai';
 
const questions = {
  intent: {
    type: 'choice',
    instructions: 'Which team should handle this support request?',
    criteria: {
      billing: 'Charges, invoices, and refunds',
      technical: 'Bugs and outages',
      account: 'Sign-in and account access',
    },
  },
} as const;
 
const result = await decide({
  model: gateway.decisionModel('typesafe-ai/jev'),
  state: 'I was charged twice and cannot sign in to request a refund.',
  questions,
  providerOptions: {
    gateway: {
      models: [
        {
          model: 'openai/gpt-6-astra',
          when: {
            question: 'intent',
            confidenceBelow: 0.6,
          },
        },
      ],
    } satisfies GatewayProviderOptions,
  },
});
 
console.log(result.answers);
console.log(result.response.modelId);
console.log(result.providerMetadata?.gateway);

const response = await fetch('https://ai-gateway.vercel.sh/v1/evaluate', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.AI_GATEWAY_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'typesafe-ai/jev',
    state: 'I was charged twice and cannot sign in to request a refund.',
    questions: {
      intent: {
        type: 'choice',
        instructions: 'Which team should handle this support request?',
        criteria: {
          billing: 'Charges, invoices, and refunds',
          technical: 'Bugs and outages',
          account: 'Sign-in and account access',
        },
      },
    },
    providerOptions: {
      gateway: {
        models: [
          {
            model: 'openai/gpt-6-astra',
            when: { question: 'intent', confidenceBelow: 0.6 },
          },
        ],
      },
    },
  }),
});
if (!response.ok) throw new Error(await response.text());
console.log(await response.json());

import json
import os
import urllib.request
 
body = {
    "model": "typesafe-ai/jev",
    "state": "I was charged twice and cannot sign in to request a refund.",
    "questions": {
        "intent": {
            "type": "choice",
            "instructions": "Which team should handle this support request?",
            "criteria": {
                "billing": "Charges, invoices, and refunds",
                "technical": "Bugs and outages",
                "account": "Sign-in and account access",
            },
        }
    },
    "providerOptions": {
        "gateway": {
            "models": [
                {
                    "model": "openai/gpt-6-astra",
                    "when": {"question": "intent", "confidenceBelow": 0.6},
                }
            ]
        }
    },
}
 
request = urllib.request.Request(
    "https://ai-gateway.vercel.sh/v1/evaluate",
    data=json.dumps(body).encode(),
    headers={
        "Authorization": "Bearer " + os.environ["AI_GATEWAY_API_KEY"],
        "Content-Type": "application/json",
    },
)
with urllib.request.urlopen(request) as response:
    print(json.load(response))

curl --fail-with-body https://ai-gateway.vercel.sh/v1/evaluate \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "typesafe-ai/jev",
  "state": "I was charged twice and cannot sign in to request a refund.",
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "Which team should handle this support request?",
      "criteria": {
        "billing": "Charges, invoices, and refunds",
        "technical": "Bugs and outages",
        "account": "Sign-in and account access"
      }
    }
  },
  "providerOptions": {
    "gateway": {
      "models": [
        {
          "model": "openai/gpt-6-astra",
          "when": { "question": "intent", "confidenceBelow": 0.6 }
        }
      ]
    }
  }
}'

confidenceBelow also matches conservatively when the primary Choice or Score answer has no finite confidence value. AI Gateway records the trigger reason as confidence_unavailable instead of silently accepting an answer it cannot assess.

Only native decision models report confidence. A language-model fallback returns one structured value per question, so its Choice and Score answers have no confidence. See Use the HTTP decision API.

A Boolean answer contains P(true), the probability that the answer is true. It does not contain Choice or Score confidence. Use the inclusive probabilityBetween range to identify an uncertain probability band:

import { gateway, type GatewayProviderOptions } from '@ai-sdk/gateway';
import { experimental_decide as decide } from 'ai';
 
const questions = {
  requestsRefund: {
    type: 'boolean',
    instructions: 'Does the customer ask for a refund?',
  },
} as const;
 
const result = await decide({
  model: gateway.decisionModel('typesafe-ai/jev'),
  state: 'I was charged twice and now the app will not load at all.',
  questions,
  providerOptions: {
    gateway: {
      models: [
        {
          model: 'openai/gpt-6-astra',
          when: {
            question: 'requestsRefund',
            probabilityBetween: [0.4, 0.6],
          },
        },
      ],
    } satisfies GatewayProviderOptions,
  },
});
 
console.log(result.answers.requestsRefund.probability);

curl --fail-with-body https://ai-gateway.vercel.sh/v1/evaluate \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "typesafe-ai/jev",
  "state": "I was charged twice and now the app will not load at all.",
  "questions": {
    "requestsRefund": {
      "type": "boolean",
      "instructions": "Does the customer ask for a refund?"
    }
  },
  "providerOptions": {
    "gateway": {
      "models": [
        {
          "model": "openai/gpt-6-astra",
          "when": {
            "question": "requestsRefund",
            "probabilityBetween": [0.4, 0.6]
          }
        }
      ]
    }
  }
}'

The condition matches when P(true) is greater than or equal to 0.4 and less than or equal to 0.6. On the TypeSafe-compatible API, the equivalent question type is Noul and its answer field is noul.

Do not use confidenceBelow with a Boolean or Noul question. AI Gateway validates condition predicates against the declared question types before running the primary model.

Leave out question to check every question of the matching type. { confidenceBelow } checks each Choice and Score answer, and { probabilityBetween } checks each Boolean and Noul answer. The condition matches when some answer crosses the threshold, so one unsure answer is enough to fall back:

{ "confidenceBelow": 0.6 }

A condition can be direct or combine other conditions:

For example, use any to escalate when either signal is uncertain:

import { gateway, type GatewayProviderOptions } from '@ai-sdk/gateway';
import { experimental_decide as decide } from 'ai';
 
const questions = {
  intent: {
    type: 'choice',
    instructions: 'Which team should handle this support request?',
    criteria: {
      billing: 'Charges, invoices, and refunds',
      technical: 'Bugs and outages',
    },
  },
  requestsRefund: {
    type: 'boolean',
    instructions: 'Does the customer ask for a refund?',
  },
} as const;
 
const result = await decide({
  model: gateway.decisionModel('typesafe-ai/jev'),
  state: 'I was charged twice and now the app will not load at all.',
  questions,
  providerOptions: {
    gateway: {
      models: [
        {
          model: 'openai/gpt-6-astra',
          when: {
            any: [
              { question: 'intent', confidenceBelow: 0.6 },
              { question: 'requestsRefund', probabilityBetween: [0.4, 0.6] },
            ],
          },
        },
        'anthropic/claude-sonnet-5',
      ],
    } satisfies GatewayProviderOptions,
  },
});

curl --fail-with-body https://ai-gateway.vercel.sh/v1/evaluate \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "typesafe-ai/jev",
  "state": "I was charged twice and now the app will not load at all.",
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "Which team should handle this support request?",
      "criteria": {
        "billing": "Charges, invoices, and refunds",
        "technical": "Bugs and outages"
      }
    },
    "requestsRefund": {
      "type": "boolean",
      "instructions": "Does the customer ask for a refund?"
    }
  },
  "providerOptions": {
    "gateway": {
      "models": [
        {
          "model": "openai/gpt-6-astra",
          "when": {
            "any": [
              { "question": "intent", "confidenceBelow": 0.6 },
              { "question": "requestsRefund", "probabilityBetween": [0.4, 0.6] }
            ]
          }
        },
        "anthropic/claude-sonnet-5"
      ]
    }
  }
}'

The trailing anthropic/claude-sonnet-5 string is an execution-error fallback. It runs only if openai/gpt-6-astra fails after the condition matched, or if both the primary and the conditional model fail to execute.

Use all to require both signals. This is the when value only:

{
  "all": [
    { "question": "intent", "confidenceBelow": 0.6 },
    { "question": "severity", "confidenceBelow": 0.7 }
  ]
}

The following policy requires two of three conditions to match:

{
  "model": "openai/gpt-6-astra",
  "when": {
    "atLeast": {
      "count": 2,
      "conditions": [
        { "question": "intent", "confidenceBelow": 0.6 },
        { "question": "severity", "confidenceBelow": 0.7 },
        {
          "question": "requestsRefund",
          "probabilityBetween": [0.4, 0.6]
        }
      ]
    }
  }
}

AI Gateway applies these structural rules:

• The conditional object must be the first models entry, and models can contain only one conditional object.
• The conditional object's model cannot be the primary model, including the primary model's fast-mode or base-tier counterpart.
• A question value, when present, must be 1 to 256 characters and name a question in the decision request.
• confidenceBelow and both probabilityBetween bounds must be finite numbers from 0 through 1.
• The probabilityBetween lower bound cannot exceed its upper bound.
• any, all, and atLeast.conditions must each contain 1 to 20 conditions.
• atLeast.count must be an integer from 1 through the number of nested conditions.
• Conditions can nest at most 5 levels deep.
• Only decision requests accept a conditional object. Other request types that include one in models are rejected.

A request that breaks any of these rules returns a 400 validation error before AI Gateway runs the primary decision.

Compound conditions short-circuit after AI Gateway can determine their result. A missing or non-finite confidence counts as a conservative match inside any, all, and atLeast.

Before any billable primary work, AI Gateway validates the condition structure and the question IDs and types it references, then checks that the conditional model exists and passes your routing rules. A language-model fallback also gets a full routing check of the providers that can serve it. It then executes the policy as follows:

1. AI Gateway runs the primary decision with the original state and every original question.
2. AI Gateway checks when against the primary answers.
3. If when does not match, AI Gateway returns the primary result unchanged.
4. If when matches, AI Gateway reruns the original state and every original question with the conditional model.
5. AI Gateway returns the complete fallback result without merging answers from the two stages.

AI Gateway does not check the condition again against the fallback result, so a request runs at most two successful decision stages.

The check that each primary answer has the shape the condition needs runs after the billed primary stage. If the provider returns a malformed answer, such as a missing answer, an answer of the wrong type, or a non-finite probability, the request fails with a 500 status. /v1/evaluate returns an internal_server_error error, the TypeSafe-compatible API returns an internal_error error, and the AI SDK throws a GatewayInternalServerError. The primary stage is still billed.

If the condition matches and the fallback stage fails, including any string fallbacks listed after the conditional object, the request fails with the fallback's error. AI Gateway does not return the primary result in that case, and the primary stage is still billed.

The final response identifies the model that produced the returned answers. In the AI SDK, read result.response.modelId. /v1/evaluate returns the final model in its top-level model field. The TypeSafe-compatible API's top-level model echoes the model ID you requested when the primary answered, and names the fallback model whenever a fallback answered.

When a condition matches, the fallback attempt in routing.modelAttempts carries triggeredBy, a list of { question, reason } entries for the direct conditions that matched. In the AI SDK, read it from result.providerMetadata?.gateway?.routing. /v1/evaluate returns it under providerMetadata.gateway.routing, and the TypeSafe-compatible API under provider_metadata.gateway.routing.

When the condition doesn't match, the conditional model doesn't appear in modelAttempts.

Provider-specific options stay at the top level of providerOptions. They carry to every stage, and only a matching provider reads its namespace. AI Gateway routing fields remain inside providerOptions.gateway.

This direct language-model fallback requests high OpenAI reasoning effort:

import { gateway, type GatewayProviderOptions } from '@ai-sdk/gateway';
import { experimental_decide as decide } from 'ai';
 
const questions = {
  quality: {
    type: 'score',
    instructions: 'Rate the response quality.',
    criteria: ['incorrect', 'partially correct', 'fully correct'],
  },
} as const;
 
const result = await decide({
  model: gateway.decisionModel('typesafe-ai/jev'),
  state: 'The response to rate.',
  questions,
  providerOptions: {
    gateway: {
      models: [
        {
          model: 'openai/gpt-6-astra',
          when: { question: 'quality', confidenceBelow: 0.7 },
        },
      ],
    } satisfies GatewayProviderOptions,
    openai: {
      reasoningEffort: 'high',
    },
  },
});

The conditional model object currently does not accept nested providerOptions. Use top-level provider options for a direct model. Use a fallback virtual model when only the fallback should get a provider order, provider-only routing, provider options, bring your own key (BYOK) credentials, an inference region, or a service tier.

A request-scoped condition can target a virtual model. The virtual model's model and routing configuration apply only to the fallback attempt:

{
  "model": "typesafe-ai/jev",
  "state": "I was charged twice and need help.",
  "questions": {
    "intent": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Charges and refunds",
        "technical": "Bugs and outages"
      }
    }
  },
  "providerOptions": {
    "gateway": {
      "models": [
        {
          "model": "vmc/high-reasoning-fallback",
          "when": { "question": "intent", "confidenceBelow": 0.6 }
        }
      ]
    }
  }
}

Configure vmc/high-reasoning-fallback with the fallback model, provider routing, and provider-specific reasoning options you want.

If the fallback virtual model has its own string fallbacks, AI Gateway tries them after its model and before any string fallbacks in the request. Its own conditional policy doesn't run, so the request still makes at most two decision stages.

A decision virtual model can store the conditional policy so every caller uses it:

{
  "modelSlug": "typesafe-ai/jev",
  "models": [
    {
      "model": "vmc/high-reasoning-fallback",
      "when": {
        "question": "intent",
        "confidenceBelow": 0.6
      }
    }
  ]
}

In the dashboard, open a decision virtual model, add the fallback model as the first entry under Model Fallbacks, and select Add Condition on it. Choose Any question or enter a request question ID, choose the predicate, and combine conditions. A virtual model cannot discover or validate the questions that future requests will send. A condition that names a question needs callers to send a matching ID and question type, which AI Gateway validates at runtime. A condition without a question works with any questions, as described in Check every question.

When a virtual model has a models policy, it replaces the request's providerOptions.gateway.models value. A request can also target a fallback virtual model from its own policy, as shown above.

POST /v1/evaluate accepts the same conditional providerOptions.gateway.models object as AI SDK decide. The HTTP API and cURL tabs above show complete requests. Provider-specific options such as openai.reasoningEffort go at the top level of providerOptions, next to gateway.

Native decision results preserve Choice and Score confidence and probability distributions when the provider returns them. A language-model fallback doesn't produce confidence: it returns one structured value per question, with no probability distribution to measure concentration from. /v1/evaluate omits confidence and probabilities from those Choice and Score answers. Boolean answers continue to return their final probability.

POST /typesafe/v1/systemone accepts the same AI Gateway extension. The official TypeSafe SDK currently forwards providerOptions at runtime but does not include the field in its static request type. Supply it through an intermediate request object:

import { TypeSafeClient } from '@typesafe-ai/sdk';
 
const client = new TypeSafeClient({
  apiKey: process.env.AI_GATEWAY_API_KEY,
  baseURL: 'https://ai-gateway.vercel.sh/typesafe',
});
 
const request = {
  model: 'typesafe-ai/jev',
  state: 'I was charged twice and cannot sign in.',
  questions: {
    intent: {
      type: 'choice' as const,
      instructions: 'Which team should handle this?',
      criteria: {
        billing: 'Charges and refunds',
        account: 'Sign-in and account access',
      },
    },
  },
  providerOptions: {
    gateway: {
      models: [
        {
          model: 'openai/gpt-6-astra',
          when: { question: 'intent', confidenceBelow: 0.6 },
        },
      ],
    },
  },
};
 
const result = await client.systemOne(request);

The TypeSafe-compatible response must satisfy TypeSafe's required fields. When a language model produces the final Choice or Score answer, AI Gateway returns confidence: 0 and probabilities: {} as unavailable sentinels. This response also shows confidence_unavailable after the primary result omitted finite confidence:

{
  "model": "openai/gpt-6-astra",
  "answers": {
    "intent": {
      "type": "choice",
      "choice": "billing",
      "confidence": 0,
      "probabilities": {}
    }
  },
  "usage": {
    "input_tokens": 641,
    "output_tokens": 57
  },
  "provider_metadata": {
    "gateway": {
      "generationId": "gen_fallback",
      "routing": {
        "originalModelId": "typesafe-ai/jev",
        "canonicalSlug": "openai/gpt-6-astra",
        "finalProvider": "openai",
        "modelAttempts": [
          {
            "canonicalSlug": "typesafe-ai/jev",
            "success": true,
            "generationId": "gen_primary",
            "usage": { "inputTokens": 347, "outputTokens": 38 }
          },
          {
            "canonicalSlug": "openai/gpt-6-astra",
            "success": true,
            "triggeredBy": [
              { "question": "intent", "reason": "confidence_unavailable" }
            ],
            "generationId": "gen_fallback",
            "usage": { "inputTokens": 294, "outputTokens": 19 }
          }
        ]
      }
    }
  }
}

On this surface, confidence: 0 and probabilities: {} mean the values are unavailable. They do not mean the fallback measured zero confidence. Native decision-model fallbacks preserve their native confidence and distributions.

Noul answers do not use the sentinel because Noul has no separate confidence or probability map. AI Gateway returns the fallback model's final noul value as P(true).

The top-level model names the model that produced the returned answers. It echoes the model ID you requested when the primary answered, and names the fallback model whenever a fallback answered, whether a condition or an execution error started it. The requested ID is a TypeSafe ID such as jev-latest, while a fallback is named by its AI Gateway slug such as openai/gpt-5-nano. To read one namespace in every case, use provider_metadata.gateway.routing.canonicalSlug, which is always an AI Gateway slug. When a condition matched, the x-ai-gateway-decision-fallback-final-model header carries the same value. routing.modelAttempts lists every model AI Gateway tried, in order. The attempt your condition started carries triggeredBy, with the questions and reasons that matched. When the condition doesn't match, the conditional model doesn't appear in modelAttempts, the same way an unused error fallback doesn't.

When a condition matches, the TypeSafe-compatible API also reports the fallback in response headers. Use them when your client only reads TypeSafe's typed response fields and ignores provider_metadata:

AI Gateway sets these headers only on responses where the condition matched. When the encoded question list would exceed 2,048 characters, it carries the questions that fit and AI Gateway adds x-ai-gateway-decision-fallback-triggering-questions-truncated: true. provider_metadata.gateway.routing always has the full list. AI Gateway lists these headers in Access-Control-Expose-Headers, so browser clients can read them. Decode the question list with decodeURIComponent, then JSON.parse:

const response = await fetch(
  'https://ai-gateway.vercel.sh/typesafe/v1/systemone',
  {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.AI_GATEWAY_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(request),
  },
);
 
if (
  response.headers.get('x-ai-gateway-decision-fallback-triggered') === 'true'
) {
  const finalModel = response.headers.get(
    'x-ai-gateway-decision-fallback-final-model',
  );
  const primaryModel = response.headers.get(
    'x-ai-gateway-decision-fallback-primary-model',
  );
  const triggeringQuestions: string[] = JSON.parse(
    decodeURIComponent(
      response.headers.get(
        'x-ai-gateway-decision-fallback-triggering-questions',
      ) ?? '[]',
    ),
  );
  console.log({ finalModel, primaryModel, triggeringQuestions });
}

AI Gateway also sends the deprecated x-ai-gateway-evaluation-fallback-* headers with identical values. Read the x-ai-gateway-decision-fallback-* headers in new code.

A triggered fallback executes and bills both the primary and fallback stages. The top-level usage and cost total both calls, and the top-level generationId identifies the generation that was returned. Each successful attempt in routing.modelAttempts carries its own generationId, usage, and cost, so you can attribute spend to each call.

The stages run sequentially, so a triggered request includes the latency of both model calls.

If the condition matches and the fallback stage fails, the request fails with the fallback's error and does not return the primary result. The primary stage is still billed.

Choose thresholds from your own results and risk tolerance. Start by recording the confidence or Boolean probability ranges associated with incorrect primary answers, then set the narrowest threshold that catches the errors worth paying the second-stage cost and latency to reassess.

• Decision
• Model fallbacks
• Provider options
• TypeSafe-compatible API

• Virtual Models