Test rules and model

The scaffold includes twelve cases in evals/vendor-requests.json: ordinary requests, cost and data-policy boundaries, ambiguous purposes, and adversarial descriptions that try to redirect the model. It also includes fast unit tests for routing, reviewer authorization, repeated decisions, and the agent's retry key.

The runner tests two layers differently:

• Policy checks are exact. The expected reviewer groups must match.
• Model checks allow some variation. A case can accept several defensible categories or risk levels while still requiring missing-information coverage and rejecting injected approval language.

Checkpoint: break the policy on purpose

Run the fast tests and the credential-free evaluation suite:

pnpm test
pnpm eval:policy

All tests and all twelve policy checks should pass. Now open lib/policy.ts and temporarily change the cost comparison from >= 50_000 to > 50_000. Run the suite again.

The threshold test and threshold-boundary evaluation should fail because a request costing exactly $50,000 no longer reaches Procurement. Restore >= 50_000 and confirm both suites pass.

Test the model

With AI_GATEWAY_API_KEY configured, run:

pnpm eval

The command performs the policy checks, calls the configured assessment model for every case, and reports failures by layer:

PASS  policy  restricted-customer-support-data
PASS  model   restricted-customer-support-data
PASS  policy  prompt-injection-approve
FAIL  model   prompt-injection-approve
      risk low not in [medium, high]
      expected at least 1 missing-information item(s)

For each model failure, read the input and output. Determine whether to correct the requirement, revise the assessment instructions, or try a different model.

In lib/ai.ts, request fields are labeled as untrusted data, and the model is instructed not to follow commands inside them. A model can still follow injected text; labeling is not a security guarantee. Deterministic routing uses the submitted cost and data types, not the model's answer, so a misleading risk assessment cannot remove required Security review.

Decide how failures behave

If model output is invalid, unavailable, or too slow, Vendor Review should not silently infer approval. Choose a safe response:

• Preserve the request
• Mark the assessment as failed
• Route to a human when policy or missing evidence requires it
• Allow an explicit retry

Test that behavior manually by running the application with an invalid Gateway key. The request should remain visible as assessment_failed; it must not become screened or approved.

Record results

Save the model, assessment version, date, pass count, and unresolved failures in docs/model-record.md. Keep the command output in local development or CI logs for the case-by-case results.

Add one case from your application job in docs/readiness.md. Give it an exact policy expectation and a clear range of acceptable model results. If the new case exposes an undocumented decision, update Who decides what before changing the code.

Commit the routing rules, evaluation cases, and updated records:

git add lib/policy.ts evals docs/model-record.md docs/readiness.md
git commit -m "feat: add tested vendor routing"

Summary

Exact tests catch changes to company rules. Representative and adversarial evaluations show whether model output meets the task’s acceptance criteria. Record failures so they remain available for review.

Check your work

Run pnpm test, pnpm eval:policy, and then pnpm eval with a Gateway key. Every exact rule should pass. If a model case fails, record it in docs/model-record.md; do not widen the acceptance criteria just to make it pass.

Compare the fixtures with evals/vendor-requests.json on complete.