Which answer does the app use?

The result can say Selected by Jev or Selected by Luna. Jev gets the first attempt. If its confidence is too low, the app asks Luna to make the same decision.

Open lib/router.ts. This branch decides whether to use Jev's answer:

if (confidence !== null && confidence >= CONFIDENCE_THRESHOLD) {
  return { ...decision, destination, model: "typesafe-ai/jev" };
}

CONFIDENCE_THRESHOLD is 0.95, or 95%. It's the minimum confidence this app accepts. When Jev returns valid confidence at or above that value, its selected team becomes the result. Otherwise, the code continues to the fallback call.

Outcome

Predict the fallback behavior from raw confidence values and verify that both models receive the same routing policy.

Hands-on exercise

Read the two numbers

Jev assigns a probability to each possible destination. That collection of numbers is its probability distribution. The result's Jev choice probability displays the entry for the team Jev selected, taken from answer.probabilities.

Jev confidence summarizes how strongly that distribution favors one answer over the others. It arrives in providerMetadata, extra information returned by the model provider. In lib/router.ts, the app reads providerMetadata.typesafe.confidence.destination. It uses Zod, a validation library, to check that the value is a number between 0 and 1.

The branch compares confidence with 0.95, using the raw value. This threshold controls when the app accepts Jev. We check whether that answer is correct by comparing it with the expected team.

Predict which model decides

For each pair of test values, predict whether the code accepts Jev's answer or calls Luna. These are artificial inputs chosen to test the code's decision rule, including whether it uses the wrong number. Assume both models return a valid destination.

Follow the second call

When Jev's confidence is below 0.95, missing, or invalid, execution continues to generateText. A Jev error also reaches that call. Find this line in its options:

prompt: JSON.stringify({ questions, state }),

The prompt is the input sent to Luna. JSON.stringify converts our question and form fields into text for that input. Luna receives the same information as Jev, without Jev's answer, and makes its own choice. Once the code checks that choice against the allowed destinations, Luna's answer becomes final.

This is the structured-output approach we discussed in Meet Jev. Find Output.object in the same call: it defines the shape Luna must return, with a destination chosen from the allowed IDs. Both approaches can choose a team. Jev's evaluate call also supplies the choice probabilities and confidence we inspect in the result.

In the UI, Selected by identifies the model that supplied the final destination. Jev's statistics remain attached to the result even if Luna chooses another team. If both models fail, the routing function rejects and the form reports a failure.

Check the policy with controlled answers

Open lib/router.test.ts. The helper mockJev(confidence, probability) supplies fixed model answers so we can force a particular branch. Find the cases for 0.95, 0.94, and does not substitute selected probability for confidence. The expect(...) lines are assertions: checks that fail the test when the result differs from what we specified. Compare them with your predictions.

Run this test file from the app folder in your second terminal:

pnpm exec vitest run lib/router.test.ts

pnpm exec runs a tool installed in this project. Here it runs Vitest, the test runner, for lib/router.test.ts. Check that the terminal reports the tests as passed.

The suite also checks 0.94999: the UI rounds it to 95.00%, but the raw value is below the threshold and calls Luna.

Include our new request

Find the test whose name begins passes identical state and criteria. It currently uses the first sample, so it doesn't exercise the cancellation sample we added.

Change it to it.each(example.samples). This runs the same test once for every sample in the array, passing the current entry to the callback as sample. Use sample.values in the call to routeSubmission.

Keep the assertions that compare the two models' inputs and exclude Jev's statistics from Luna's prompt. The Solution section provides the full replacement.

The fake Luna model always returns contact_triage in this test. That lets us check whether the router uses the fallback's answer, including when it differs from Jev's choice.

Try It

Run the focused suite again and check that the input-preservation test passes for each of the four samples. Inspect fails when both providers fail too: its assertion expects the routing function to reject.

Then explain this possible result without running a model: Jev chooses refunds with confidence 0.80, and Luna chooses account access. Which team is final, and whose confidence will the UI show? Check your explanation against the solution.

If something goes wrong

The new test still checks one request: make sure the call uses sample.values. Changing the test heading alone leaves the old submission input in place.

Your prediction follows the larger number: locate the metadata read in lib/router.ts. selectedProbability is saved for inspection; the acceptance branch uses confidence.

Checkpoint

Save lib/router.test.ts and your four predictions in routing-notes.md. Explain why a high selected-option probability does not change the confidence comparison.

Done-When

• You can explain each prediction using the acceptance branch.
• The input-preservation test runs once per contact sample and passes.
• You can identify the final destination when the models disagree.
• You can locate the test for failure of both providers.

Solution

The predictions, in order, are Jev, Luna, Jev, Luna. The third row accepts Jev because confidence exceeds the threshold, even though its selected-option probability is lower. Missing confidence sends the fourth row to Luna.

In the disagreement example, account access is final because Luna supplies it. The displayed 0.80 confidence still describes Jev's refund choice.

Replace the single input-preservation test with this block inside the existing describe("routing policy", ...). It reuses the file's imports and mock helpers:

it.each(example.samples)(
  "passes identical state and criteria to the fallback for $label without Jev's answer",
  async (sample) => {
    const original = mockJev(0.8).model;
    const evaluateCall = vi.fn(original.doEvaluate);
    const jev = new Experimental_EvaluationMockModelV4({
      doEvaluate: evaluateCall,
    });
    const luna = mockLuna("contact_triage");
    const result = await routeSubmission(example, sample.values, {
      jev,
      luna,
    });
    const [[evaluation]] = evaluateCall.mock.calls;
    const prompt = JSON.stringify(luna.doGenerateCalls[0].prompt);
    expect(prompt).toContain(
      JSON.stringify(
        JSON.stringify({
          questions: evaluation.questions,
          state: evaluation.state,
        })
      ).slice(1, -1)
    );
    expect(prompt).not.toContain("probabilities");
    expect(prompt).not.toContain("confidence");
    expect(result.destination.id).toBe("contact_triage");
  }
);

vi.fn records calls to the fake Jev model. The test reads that record to see what Jev received, then checks for the same question and state in Luna's recorded input. The nested JSON.stringify calls handle the escaped JSON inside that recorded prompt.

The last assertion checks that the router returns Luna's chosen team. With it.each, we now check this path for every contact sample, including our cancellation request.