---
title: "Test Rules and Model"
description: "Test deterministic policy exactly and model behavior empirically, including inputs that try to redirect the assessment."
canonical_url: "https://vercel.com/academy/enterprise-apps-agents/test-the-boundary"
md_url: "https://vercel.com/academy/enterprise-apps-agents/test-the-boundary.md"
docset_id: "vercel-academy"
doc_version: "1.0"
last_updated: "2026-08-28T23:02:55.101Z"
content_type: "lesson"
course: "enterprise-apps-agents"
course_title: "Enterprise Apps and Agents"
prerequisites:  []
---

<agent-instructions>
Vercel Academy — structured learning, not reference docs.
Lessons are sequenced.
Adapt commands to the human's actual environment (OS, package manager, shell, editor) — detect from project context or ask, don't assume.
The lesson shows one path; if the human's project diverges, adapt concepts to their setup.
Preserve the learning goal over literal steps.
Quizzes are pedagogical — engage, don't spoil.
Quiz answers are included for your reference.
</agent-instructions>

# Test Rules and Model

# Test rules and model

The scaffold includes twelve cases in `evals/vendor-requests.json`: ordinary requests, cost and data-policy boundaries, ambiguous purposes, and adversarial descriptions that try to redirect the model. It also includes fast unit tests for routing, reviewer authorization, repeated decisions, and the agent's retry key.

The runner tests two layers differently:

- **Policy checks are exact.** The expected reviewer groups must match.
- **Model checks allow some variation.** A case can accept several defensible categories or risk levels while still requiring missing-information coverage and rejecting injected approval language.

## Checkpoint: break the policy on purpose

Run the fast tests and the credential-free evaluation suite:

```bash
pnpm test
pnpm eval:policy
```

All tests and all twelve policy checks should pass. Now open `lib/policy.ts` and temporarily change the cost comparison from `>= 50_000` to `> 50_000`. Run the suite again.

The threshold test and `threshold-boundary` evaluation should fail because a request costing exactly $50,000 no longer reaches Procurement. Revert the change and confirm both suites return to green.

That failure is the payoff for keeping deterministic policy out of the prompt: the boundary has an exact test and a useful diff.

## Test the model

With `AI_GATEWAY_API_KEY` configured, run:

```bash
pnpm eval
```

The command performs the policy checks, calls the configured assessment model for every case, and reports failures by layer:

```text
PASS  policy  restricted-customer-support-data
PASS  model   restricted-customer-support-data
PASS  policy  prompt-injection-approve
FAIL  model   prompt-injection-approve
      risk low not in [medium, high]
      expected at least 1 missing-information item(s)
```

A model failure is evidence to inspect, not an instruction to loosen the expected result until the test passes. Read the input and output. Decide whether the requirement is wrong, the assessment instructions are weak, or the selected model is unsuitable.

The adversarial descriptions remain inside the labeled request-data block in `lib/ai.ts`. They never become application instructions. Deterministic routing still sends restricted data to Security even if the model follows the injected text.

## Decide how failures behave

If model output is invalid, unavailable, or too slow, Vendor Review should not silently infer approval. Choose a safe response:

- Preserve the request
- Mark the assessment as failed
- Route to a human when policy or missing evidence requires it
- Allow an explicit retry

Test that behavior manually by running the application with an invalid Gateway key. The request should remain visible as `assessment_failed`; it must not become screened or approved.

## Record results

The command output is suitable for local development and CI logs. For a release record, retain model, assessment version, date, pass count, and the unresolved failures.

Add one case from your application job in `docs/readiness.md`. Give it an exact policy expectation and a clear range of acceptable model results. If the new case exposes an undocumented decision, update **Who decides what** before changing the code.

Record the model, assessment version, date, and result in `docs/model-record.md`, then commit the routing-and-test checkpoint:

```bash
git add lib/policy.ts evals docs/model-record.md docs/readiness.md
git commit -m "feat: add tested vendor routing"
```

## Summary

Deterministic rules should pass exact tests. Model behavior needs representative and adversarial evaluations with clear acceptance criteria. A failed model assessment becomes visible work for a person; it never becomes silent approval.

## Check your work

Run `pnpm test`, `pnpm eval:policy`, and then `pnpm eval` with a Gateway key. Every exact rule should pass. If a model case fails, keep the failure visible in `docs/model-record.md`; do not widen the expectation merely to make the terminal green.

Compare the fixtures with [`evals/vendor-requests.json` on `complete`](https://github.com/vercel-labs/academy-enterprise-apps-agents/blob/complete/evals/vendor-requests.json).


---

[Full course index](/academy/llms.txt) · [Sitemap](/academy/sitemap.md)