An Introduction to Evals

Evaluations (or "evals") are systematic tests that measure how well AI models perform at specific tasks. Like traditional unit, integration, and end-to-end tests, evals ensure your code remains reliable and stable. However, they differ in one substantial way: the underlying system being tested is non-deterministic due to the fact that outputs can vary slightly or significantly between runs. Evals are designed specifically to test systems robustly when outputs aren't perfectly consistent.

This guide will walk you through the fundamentals of building and implementing effective evaluation frameworks for AI applications.

For many AI apps, developers run a few examples and check if the outputs "feel right" before shipping. It's all about the "vibes." Unfortunately, this vibe-based approach doesn't scale, and when deploying AI models in production, you need confidence that they'll perform consistently and safely. Evals help you:

Compare models and prompts. Should you use GPT-5 or Claude? Will adding examples to your prompt improve accuracy? Without evals, these decisions rely on guesswork and anecdotal evidence. With a solid evaluation framework, you can run the same test suite across different configurations and make data-driven decisions about which approach actually performs better for your specific use case.

Catch regressions early. AI systems are constantly evolving. You might update your prompt, switch models, or modify your system architecture. Each change risks breaking functionality that previously worked. Evals act as a safety net, automatically flagging when changes degrade performance on known test cases, preventing you from shipping regressions to production.

Identify edge cases. Real-world usage always surfaces scenarios you didn't anticipate. A comprehensive eval suite helps you discover where your system breaks down, whether it's ambiguous inputs, unusual formatting, edge cases in your domain. Once identified, these failures become test cases, ensuring you don't regress as you improve the system.

Every eval system consists of three main building blocks: the dataset, the runner, and the scorer.

The dataset is your collection of test cases. This includes the inputs and expected outputs that define what success looks like. A well-constructed dataset is the foundation of an effective eval system.

Your dataset should mirror real-world usage. Start with actual production examples, edge cases you've discovered, and failure modes you want to prevent. Quality beats quantity. 20 well-chosen test cases that cover your core use cases are more valuable than 200 random examples.

The runner (also called a "harness" or "executor") is the orchestration layer that executes your test cases. It feeds inputs to your AI system, collects outputs, and manages the evaluation workflow.

A good runner is model-agnostic, meaning you can swap out GPT-4 for Claude or change your prompt without rewriting your entire eval system. This flexibility is key for comparing different approaches.

The scorer (or "grader") evaluates how well the actual outputs match expected results. This is where evals diverge most significantly from traditional testing. Instead of exact matches, you need methods that account for the variability in AI outputs.

Common approaches include exact matching (for structured outputs like JSON), semantic similarity (checking if outputs mean the same thing even with different wording), LLM-as-judge (using another AI to evaluate quality), and custom metrics tailored to your domain.

Given an AI agent that classifies customer support emails as “billing,” “technical,” or “general,” let’s walk through the building of an eval system.

We start with a dataset that defines what good performance looks like. Each test case includes an input, expected output, and metadata.

1
const dataset = [
2
  {
3
    input: "I was charged twice for my subscription this month.",
4
    expected: "billing",
5
    difficulty: "easy"
6
  },
7
  {
8
    input: "The app crashes when I export to CSV.",
9
    expected: "technical",
10
    difficulty: "easy"
11
  },
12
  {
13
    input: "Love your product! Thanks for the great service.",
14
    expected: "general",
15
    difficulty: "easy"
16
  },
17
  {
18
    input: "Payment failed but I see a pending charge. Is this a bug?",
19
    expected: "billing",
20
    difficulty: "hard"
21
  }
22
];

Notice the mix of straightforward cases and edge cases. The hard example mentions both payment issues and potential bugs, exactly the kind of ambiguity you'll encounter in production.

After wiring up the dataset, the runner executes your test cases and collects results. Here's the core logic:

1
import { generateObject } from "ai";
2
import { z } from "zod";
3

4
const ClassificationSchema = z.object({
5
  category: z.enum(["billing", "technical", "general"]),
6
  confidence: z.number().min(0).max(1)
7
});
8

9
interface ClassifyEmailInput {
10
  model: string;
11
  email: string;
12
}
13

14
export async function classifyEmail({ model, email }: ClassifyEmailInput) {
15
  const { object } = await generateObject({
16
    model,
17
    schema: ClassificationSchema,
18
    prompt: `Classify the following customer support email into one of three categories: "billing", "technical", or "general". 
19
            Return a JSON object with both "category" and "confidence" (0.0–1.0).
20

21
            Email:
22
            ${email}
23
            `
24
  });
25

26
  return object;
27
}
28

29
async function runEval(modelId: string) {
30
    const results = [];
31
    for (const testCase of dataset) {
32
      const { category, confidence } = await classifyEmail({
33
        model: modelId,
34
        email: testCase.input
35
      });
36
      results.push({
37
        input: testCase.input,
38
        expected: testCase.expected,
39
        actual: category,
40
        confidence: confidence,
41
        passed: category === testCase.expected
42
      });
43
    }
44
    return results;
45
}

In the above code snipped we leverage the AI SDK to keep the runner as model-agnostic, letting you swap out modelId to test GPT-5, Claude, or any other model without changing your evaluation logic.

Putting it all together, you can now compare different models systematically via scorer:

1
interface EvalResult {
2
  input: string;
3
  expected: string;
4
  actual: string;
5
  confidence: number;
6
  passed: boolean;
7
}
8

9
interface Metrics {
10
  accuracy: number;
11
}
12

13
export function calculateMetrics(results: EvalResult[]): Metrics {
14
  const total = results.length;
15
  const passed = results.filter(r => r.passed).length;
16
  const accuracy = total > 0 ? passed / total : 0;
17

18
  return { accuracy };
19
}
20

21
const gpt5Results = await runEval('openai/gpt-5');
22
const gpt5Metrics = calculateMetrics(gpt5Results);
23

24
const claudeResults = await runEval('anthropic/claude-sonnet-4.5');
25
const claudeMetrics = calculateMetrics(claudeResults);
26

27
console.log('GPT-5 Accuracy:', gpt5Metrics.accuracy);     
28
console.log('Claude Accuracy:', claudeMetrics.accuracy);  

In this example, the calculateMetrics function leverages exact matching to grade the output from the runner.

With this foundation, you can systematically improve your AI system—measuring every change, catching regressions early, and making decisions based on evidence rather than intuition.

• Use the AI SDK to write eval code once and test across multiple models by simply changing the model identifier
• Leverage Vercel AI Gateway for automatic caching, built-in observability, and provider fallbacks when running eval suites
• Implement monitoring with tools like Braintrust with automatic scoring and experiment tracking
• Learn from Eval-Driven Development for real-world patterns on integrating evals into your development process