---
title: Evaluation
product: vercel
url: /docs/ai-gateway/modalities/evaluation
canonical_url: "https://vercel.com/docs/ai-gateway/modalities/evaluation"
last_updated: 2018-10-20
type: conceptual
prerequisites:
  - /docs/ai-gateway/modalities
  - /docs/ai-gateway
related:
  []
summary: Evaluate shared state against typed questions and get back structured choices, scores, and boolean probabilities through Vercel AI Gateway.
install_vercel_plugin: npx plugins add vercel/vercel-plugin
---

# Evaluation

Evaluate a piece of shared state against typed questions and get structured answers back. Evaluation models return choices, scores, and boolean probabilities rather than free-form text, which makes them a fit for classification, routing, rubric-based assessment, and automated verification.


<!-- docsgraph:related -->
## Related pages

> **For AI agents:** Follow these links to understand how this page connects to the rest of the Vercel ecosystem. For the full cross-link map (inbound, outbound, prerequisites, and semantic neighbors), see the .graph.md link below.

- [TypeSafe AI's Jev now available on AI Gateway](https://vercel.com/changelog/typesafe-ai-jev-now-available-on-ai-gateway?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=related)
- [Evaluation](https://ai-sdk.dev/docs/ai-sdk-core/evaluation?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=related)
- [An Introduction to Evals](https://vercel.com/kb/guide/an-introduction-to-evals?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=related) — Evaluations test model and agent outputs to ensure they meet the standards and requirements you specify.
- [experimental_evaluate](https://ai-sdk.dev/docs/reference/ai-sdk-core/evaluate?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=related)
- [Automatic Model Selection](https://eve.dev/docs/guides/evaluate?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=related) — Choose an agent model from the current request with an AI SDK evaluation model.
- [Eval-driven development: Build better AI faster](https://vercel.com/blog/eval-driven-development-build-better-ai-faster?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=related)
- [AI Gateway: Production-ready reliability for your AI apps](https://vercel.com/blog/ai-gateway-is-now-generally-available?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=related)
- [AI Gateway SDKs and APIs](https://vercel.com/docs/ai-gateway/sdks-and-apis?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=related) — Connect to AI Gateway with the AI SDK, Python, REST, or compatible OpenAI, Anthropic Messages, OpenResponses, and Cohere

Full cross-link map for this page: [/docs/ai-gateway/modalities/evaluation.graph.md](/docs/ai-gateway/modalities/evaluation.graph.md?from=related&source_path=%2Fdocs%2Fai-gateway%2Fmodalities%2Fevaluation&source_site=vercel-docs&relationship=graph)
<!-- /docsgraph:related -->

Several questions can be answered in parallel within a single request, against the same state.

To see which models AI Gateway supports for evaluation, use the **Evaluation** filter at the [AI Gateway Models page](/ai-gateway/models?capabilities=evaluation).

> **💡 Note:** Evaluation is available through the AI SDK only. It is not supported through
> the OpenAI-compatible, Anthropic-compatible, or Cohere-compatible endpoints.
> It requires AI SDK 7 or later.

## Basic usage

```typescript filename="app/api/evaluate/route.ts" {5-13}
import { experimental_evaluate as evaluate } from 'ai';

export async function GET() {
  const result = await evaluate({
    model: 'typesafe-ai/jev',
    state: 'The support agent issued a full refund to the customer.',
    questions: {
      refunded: {
        type: 'boolean',
        instructions: 'Was a refund issued?',
      },
    },
  });

  return Response.json(result.answers);
}
```

Each key in `questions` becomes a key in `answers`:

```typescript
// result.answers
{
  refunded: { type: 'boolean', probability: 0.99 }
}
```

## Question types

### Boolean

Returns a probability between 0 and 1. Supply `criteria` to define what the true and false cases mean.

```typescript
const result = await evaluate({
  model: 'typesafe-ai/jev',
  state: 'The build failed with exit code 1.',
  questions: {
    passed: {
      type: 'boolean',
      instructions: 'Did the build succeed?',
      criteria: {
        true: 'exit code 0',
        false: 'any non-zero exit code',
      },
    },
  },
});

// { passed: { type: 'boolean', probability: 0.01 } }
```

### Choice

Picks one option from a named set. `criteria` is a record of option names to descriptions, and the answer carries both the selected `choice` and the probability of each option.

```typescript
const result = await evaluate({
  model: 'typesafe-ai/jev',
  state: 'My card was charged twice for one order.',
  questions: {
    route: {
      type: 'choice',
      instructions: 'Route this support ticket.',
      criteria: {
        billing: 'payment or charge problems',
        shipping: 'delivery problems',
        technical: 'application bugs',
      },
    },
  },
});

// {
//   route: {
//     type: 'choice',
//     choice: 'billing',
//     probabilities: { billing: 1, shipping: 0, technical: 0 },
//   },
// }
```

### Score

Rates the state along an ordered scale. `criteria` is an array of at least two labels, ordered lowest to highest. The answer is an interpolated `score` plus the probability of each rung.

```typescript
const result = await evaluate({
  model: 'typesafe-ai/jev',
  state: 'The PR adds tests, updates docs, and has a clear description.',
  questions: {
    quality: {
      type: 'score',
      instructions: 'Rate the quality of this pull request.',
      criteria: [
        'poor: no tests or docs',
        'fair: partial coverage',
        'good: tests and docs',
        'excellent: tests, docs, and clear rationale',
      ],
    },
  },
});

// {
//   quality: {
//     type: 'score',
//     score: 2.97,
//     probabilities: { '0': 0, '1': 0, '2': 0.02, '3': 0.98 },
//   },
// }
```

## Multiple questions in one request

Questions of different types can share a single state, and are answered in one round trip.

```typescript filename="app/api/triage/route.ts" {7-24}
import { experimental_evaluate as evaluate } from 'ai';

export async function GET() {
  const result = await evaluate({
    model: 'typesafe-ai/jev',
    state: 'I cannot log in, and I also want a refund for last month.',
    questions: {
      authIssue: {
        type: 'boolean',
        instructions: 'Is there a login problem?',
      },
      wantsRefund: {
        type: 'boolean',
        instructions: 'Is a refund requested?',
      },
      urgency: {
        type: 'score',
        instructions: 'How urgent is this ticket?',
        criteria: ['low', 'medium', 'high'],
      },
    },
  });

  return Response.json(result.answers);
}
```

## Structured state

`state` accepts a string, an object, or an array, so you can pass structured records or a message history directly without serializing them yourself.

```typescript
const result = await evaluate({
  model: 'typesafe-ai/jev',
  state: {
    order: { id: 'A-1', total: 42.5, status: 'refunded' },
    agent: 'bot-7',
  },
  questions: {
    refunded: {
      type: 'boolean',
      instructions: 'Is the order refunded?',
    },
  },
});
```

## Gateway provider instance

If you're using the Gateway provider instance, specify evaluation models with `gateway.evaluationModel(...)`.

```typescript filename="app/api/evaluate/route.ts" {2,6}
import { experimental_evaluate as evaluate } from 'ai';
import { gateway } from '@ai-sdk/gateway';

export async function GET() {
  const result = await evaluate({
    model: gateway.evaluationModel('typesafe-ai/jev'),
    state: 'The support agent issued a full refund to the customer.',
    questions: {
      refunded: {
        type: 'boolean',
        instructions: 'Was a refund issued?',
      },
    },
  });

  return Response.json(result.answers);
}
```

## Usage and pricing

Evaluation requests report token usage like any other model, and are billed from the model's per-token rates. Check the [AI Gateway Models page](/ai-gateway/models?capabilities=evaluation) for the rates on a specific model, since some evaluation models price input tokens only.

```typescript
const result = await evaluate({
  model: 'typesafe-ai/jev',
  state: 'The support agent issued a full refund.',
  questions: {
    refunded: { type: 'boolean', instructions: 'Was a refund issued?' },
  },
});

console.log(result.usage);
// { inputTokens: 283, outputTokens: 21 }
```


---

[View full sitemap](/docs/sitemap)