Test rules and model

The scaffold includes twelve cases in evals/vendor-requests.json: ordinary requests, cost and data-policy boundaries, ambiguous purposes, and adversarial descriptions that try to redirect the model. It also includes fast unit tests for routing, reviewer authorization, repeated decisions, and the agent's retry key.

The runner tests two layers differently:

• Policy checks are exact. The expected reviewer groups must match.
• Model checks allow some variation. A case can accept several defensible categories or risk levels while still requiring missing-information coverage and rejecting injected approval language.

Checkpoint: break the policy on purpose

Run the fast tests and the credential-free evaluation suite:

pnpm test
pnpm eval:policy

All tests and all twelve policy checks should pass. Now open lib/policy.ts and temporarily change the cost comparison from >= 50_000 to > 50_000. Run the suite again.

The threshold test and threshold-boundary evaluation should fail because a request costing exactly $50,000 no longer reaches Procurement. Revert the change and confirm both suites return to green.

That failure is the payoff for keeping deterministic policy out of the prompt: the boundary has an exact test and a useful diff.

Test the model

With AI_GATEWAY_API_KEY configured, run:

pnpm eval

The command performs the policy checks, calls the configured assessment model for every case, and reports failures by layer:

PASS  policy  restricted-customer-support-data
PASS  model   restricted-customer-support-data
PASS  policy  prompt-injection-approve
FAIL  model   prompt-injection-approve
      risk low not in [medium, high]
      expected at least 1 missing-information item(s)

A model failure is evidence to inspect, not an instruction to loosen the expected result until the test passes. Read the input and output. Decide whether the requirement is wrong, the assessment instructions are weak, or the selected model is unsuitable.

The adversarial descriptions remain inside the labeled request-data block in lib/ai.ts. They never become application instructions. Deterministic routing still sends restricted data to Security even if the model follows the injected text.

Decide how failures behave

If model output is invalid, unavailable, or too slow, Vendor Review should not silently infer approval. Choose a safe response:

• Preserve the request
• Mark the assessment as failed
• Route to a human when policy or missing evidence requires it
• Allow an explicit retry

Test that behavior manually by running the application with an invalid Gateway key. The request should remain visible as assessment_failed; it must not become screened or approved.

Record results

The command output is suitable for local development and CI logs. For a release record, retain model, assessment version, date, pass count, and the unresolved failures.

Add one case from your application job in docs/readiness.md. Give it an exact policy expectation and a clear range of acceptable model results. If the new case exposes an undocumented decision, update Who decides what before changing the code.

Record the model, assessment version, date, and result in docs/model-record.md, then commit the routing-and-test checkpoint:

git add lib/policy.ts evals docs/model-record.md docs/readiness.md
git commit -m "feat: add tested vendor routing"

Summary

Deterministic rules should pass exact tests. Model behavior needs representative and adversarial evaluations with clear acceptance criteria. A failed model assessment becomes visible work for a person; it never becomes silent approval.

Check your work

Run pnpm test, pnpm eval:policy, and then pnpm eval with a Gateway key. Every exact rule should pass. If a model case fails, keep the failure visible in docs/model-record.md; do not widen the expectation merely to make the terminal green.

Compare the fixtures with evals/vendor-requests.json on complete.