GPT-6 Sol can answer typed questions through AI SDK's experimental evaluation API. Create an OpenAI evaluation model, supply the evidence to assess, and define the answers your application needs. The result can classify a proposed response, score it against a rubric, or estimate whether a statement is true.
Copy link to headingWhat does evaluation mean in the AI SDK?
The AI SDK evaluation API lets application code ask named questions about shared input, called state. For example, a support application could assess a draft reply against the customer's request and the relevant policy before showing it to a reviewer.
The evaluation happens during the application's workflow. Measuring how accurately Sol performs that evaluation requires a separate comparison with examples that people have already reviewed.
The OpenAI evaluation adapter uses the Responses API and structured output to produce three answer types:
Choice and Score answers from this adapter do not include probability distributions. Boolean probabilities are generated estimates, so a result of 0.9 should not be read as proof that the model is correct nine times out of ten on your task.
Copy link to headingWhen is Sol useful for an evaluation step?
Sol is designed for complex coding and agent workflows. It is worth testing when a decision requires interpreting several pieces of evidence, such as deciding whether a draft explanation accurately describes a code change or follows a policy with exceptions.
Consider a customer asking whether an unused purchase qualifies for a return. The proposed reply says the refund has already been processed. Even if the customer is eligible, the policy alone cannot support a claim that money has been returned. An evaluation can separate whether the reply addresses the request from whether its factual claims are supported.
Supply the customer message, the applicable policy, and the draft reply together. If the application has a payment record, include it as a distinct source. Keeping those inputs separate helps define what the model should judge and gives a reviewer a way to investigate a disputed result.
Copy link to headingHow do you evaluate a draft reply with Sol?
Copy link to heading1. Install the AI SDK and OpenAI provider
In a TypeScript application, install packages that include the experimental evaluation API:
npm install ai @ai-sdk/openaiConfigure OPENAI_API_KEY as an environment variable available to the code that makes the request. These examples use the OpenAI provider directly. Its evaluationModel factory selects the Responses integration, so the model ID is gpt-6-sol without a Gateway prefix.
The evaluation API is experimental and can change in patch releases. Keep a lockfile and check the evaluation contract when updating the SDK, particularly if your application depends on specific answer fields.
Copy link to heading2. Define the evidence and questions
Put the shared example in src/reply-review.ts. The policy below is sample application data, and the questions describe how to assess a reply against it.
export const reviewState = { policy: 'Unused purchases can be returned within 30 days of delivery.', customerMessage: 'My order arrived 12 days ago and is unopened. Can I return it?', draftReply: 'Your purchase qualifies for a return. We have processed your refund.',};
export const reviewQuestions = { disposition: { type: 'choice' as const, instructions: 'Assess the draft reply using only the supplied evidence.', criteria: { ready: 'The reply answers the request and all factual claims are supported.', revise: 'The reply contains an unsupported claim, contradicts the evidence, or omits part of the request.', needs_review: 'The request or applicable policy is too ambiguous to assess whether the reply answers it.', }, }, coverage: { type: 'score' as const, instructions: 'How fully does the reply address the customer request?', criteria: [ 'Does not address the request', 'Addresses part of the request', 'Addresses the full request', ], }, unsupportedClaim: { type: 'boolean' as const, instructions: 'Does the reply make a factual claim not supported by the supplied evidence?', criteria: { true: 'At least one claim lacks support or contradicts the evidence.', false: 'Every factual claim is supported by the evidence.', }, },};disposition provides a recommendation for the review workflow, while coverage measures how much of the request the reply addresses. unsupportedClaim focuses on factual support. Coverage alone should not determine whether a reply is ready because it may address the whole request and still make an unsupported promise.
The OpenAI adapter evaluates these questions together in one prompt. If you change a question or its criteria, rerun the whole set against your reviewed examples to check whether the judgments remain useful.
Copy link to heading3. Call Sol and read the named answers
Import the shared inputs from a module that runs in your application's backend:
import { openai } from '@ai-sdk/openai';import { experimental_evaluate as evaluate } from 'ai';import { reviewQuestions, reviewState } from './reply-review';
const result = await evaluate({ model: openai.evaluationModel('gpt-6-sol'), state: reviewState, questions: reviewQuestions, providerOptions: { openai: { reasoningEffort: 'medium' }, },});
console.log(result.answers.disposition.choice);console.log(result.answers.coverage.score);console.log(result.answers.unsupportedClaim.probability);Named answers appear under answers, with disposition.choice containing one of the three configured option keys. The coverage score falls between 0 and 2 for this rubric and may be fractional. unsupportedClaim.probability estimates the truth of that specific statement, rather than confidence in the entire evaluation.
Sol supports reasoning effort settings from none through max. The evaluation adapter requests no reasoning by default, so this example explicitly selects medium. Compare effort settings on cases that require different amounts of interpretation before choosing a default for your workload.
Copy link to headingHow should your application use the result?
Display the disposition beside the original evidence and draft in the review interface. Route revise results back to the author for correction, and use needs_review to request an additional record or a person's judgment. The ready option means the model found the reply acceptable under the supplied criteria. Sending it to the customer remains a separate operation.
Keep the supporting fields visible when they disagree. If the disposition is ready but the unsupported-claim estimate is high, the reviewer needs to see that conflict. Decide how your application handles such combinations using labeled examples, and store the question definitions with the evaluation so you can reproduce what was asked.
Avoid converting the coverage score into a probability. Its scale describes positions on your rubric, and changing the rubric changes the score's meaning. Adding a fourth level, for example, changes the upper bound from 2 to 3.
Copy link to headingCan I use other OpenAI models with AI SDK evaluation?
Yes. The OpenAI evaluation adapter works with models that support its structured-output integration. GPT-6 Astra and GPT-6 Luna are alternatives you can compare using the same evidence and questions.
Copy link to headingUse Astra for complex reasoning across supplied evidence
Astra is a candidate when the review involves difficult reasoning across several records. Include those records in the shared state before making the call, since this evaluation assesses the supplied evidence without retrieving additional information.
import { openai } from '@ai-sdk/openai';import { experimental_evaluate as evaluate } from 'ai';import { reviewQuestions, reviewState } from './reply-review';
const result = await evaluate({ model: openai.evaluationModel('gpt-6-astra'), state: reviewState, questions: reviewQuestions, providerOptions: { openai: { reasoningEffort: 'high' }, },});
console.log(result.answers.disposition.choice);Astra does not support reasoning effort none. Without an explicit override, the evaluation wrapper requests none, but the OpenAI provider removes that setting and returns an unsupported-setting warning. Set a supported effort explicitly so the evaluation uses the level you intend and avoids the warning. high is an example for a demanding review; it is not required for every Astra evaluation.
Copy link to headingUse Luna for focused evaluations repeated across many inputs
Luna is a candidate for repeated checks with a narrow rubric, such as determining whether replies address a clearly stated request. Include ambiguous and incomplete examples when testing it, since routine successes can conceal weaknesses on the cases that need review.
import { openai } from '@ai-sdk/openai';import { experimental_evaluate as evaluate } from 'ai';import { reviewQuestions, reviewState } from './reply-review';
const result = await evaluate({ model: openai.evaluationModel('gpt-6-luna'), state: reviewState, questions: reviewQuestions, providerOptions: { openai: { reasoningEffort: 'none' }, },});
console.log(result.answers.disposition.choice);Luna supports none, and you can raise the effort if testing shows that reasoning improves the judgments you need. These examples demonstrate different configurations. For a comparison that isolates the model choice, use the same supported effort setting across all three, then tune each model separately.
Copy link to headingWhat happens if an evaluation cannot return usable answers?
The OpenAI adapter fails the evaluation call when it receives a refusal, truncated output, or invalid answers. Your application should record that failure separately from a completed evaluation whose disposition is needs_review. The latter is a judgment about the supplied evidence; the former means you have no usable evaluation result.
Use abortSignal when a request needs a deadline or cancellation, and configure maxRetries for transient failures. Retries can recover from temporary provider errors; missing evidence needs to be added to state before another evaluation can assess it. Preserve the draft and offer another review path if the call fails.
Evaluation returns one complete result and does not stream answers. An array passed as state is shared context for one evaluation, such as a conversation history. Evaluate unrelated replies separately so each decision has an unambiguous subject.
Copy link to headingHow do you check whether the evaluations are useful?
Build a set of draft replies with expected dispositions established by reviewers. Include cases where the reply is fluent but unsupported, as well as cases where the policy itself is incomplete. Track false approvals separately from unnecessary revision requests because those mistakes have different consequences for the support team.
Compare the model's judgments with those labels after changing the model, reasoning effort, or question definitions. Record request duration and token usage alongside the decisions to assess the cost of running the check across your expected volume. Keep a separate test for how application code responds to each answer and to failures; correct branching does not prove that the model chose the right answer.
If your application needs native distributions over a defined set of answers, Jev with AI SDK provides another evaluation path. Jev's native probabilities and the OpenAI adapter's generated Boolean estimates have different origins. Test any acceptance threshold with the provider and task you intend to use.
Copy link to headingFrequently asked questions
Copy link to headingDoes this benchmark GPT-6 Sol?
No. These examples use Sol to judge supplied application data. Benchmarking those judgments requires expected answers and a process for comparing the model's results with them.
Copy link to headingDoes Sol return a confidence score for every evaluation answer?
No. The OpenAI adapter returns Choice and Score judgments without probability distributions. Boolean questions return generated estimates of the probability that a statement is true, which need testing before use as decision thresholds.
Copy link to headingCan I disable reasoning for every GPT-6 model?
No. Sol and Luna accept none, while Astra supports reasoning levels starting at low. Set Astra's effort explicitly to avoid the provider's unsupported-setting warning.
Copy link to headingCan I evaluate a reply with an older OpenAI model?
Yes, if the model supports the OpenAI adapter's structured-output integration. Check its supported reasoning settings and compare its judgments against reviewed examples before using it in the workflow.