Measure adoption, quality, and cost

Request volume alone cannot show that Vendor Review improves the review process. Return to the baseline and target in docs/readiness.md.

Use the traffic and test requests available during the course. Record the reporting period and identify sample requests so readers do not mistake course activity for employee adoption. Use this first report to check that you can collect the measurements and set the next reporting period.

Adoption

Use Observability to inspect traffic, function errors, and external calls. Some route-level latency and path breakdowns require Observability Plus; record which measurements your plan exposes.

Use the application's persisted records to measure submitted requests, active users or teams where identity permits, completion rate, and time from submission to decision. Count completed workflows separately from page views.

Reliability

Track HTTP errors, assessment failures, workflow failures, decision conflicts, and latency. Inspect one request across application logs and its workflow run. Record how an operator moves from an alert to the affected decision record.

If your organization has access to the Security Dashboard, inspect its organization-wide findings alongside these project-level checks. The Security Dashboard is in Private Beta and surfaces security posture across accounts and projects, including missing MFA, shared secrets, and long-lived credentials. Record it as demonstrated or planned unless you actually inspected your organization there.

AI quality

Rerun the evaluation set from Make AI Decisions Dependable. Track deterministic-policy test failures separately from model-assessment failures. Model or prompt changes should show their evaluation result beside the deployed assessment version.

Cost

Use AI Gateway usage to attribute requests, tokens, and cost by model, project, and credential. Estimate cost per completed vendor request, not just cost per model call. Add a team, project, or API-key budget with enough headroom for expected use and document what happens when it is reached.

Build the first operating report

Add docs/operating-report.md:

Reporting period:
Requests submitted:
Requests completed:
Median and p90 decision time:
Assessment failures:
Workflow failures:
Evaluation pass rate:
AI cost per completed request:
Budget and owner:
Known gaps and who will respond:

Enter the results from your course run. Mark any measurement you could not collect as unavailable and name the work needed to collect it.

Contain and roll back a harmful change

Containment stops additional harm; rollback restores a known-good application version. Add this sequence under Recovery in docs/operating-report.md:

1. Stop new submissions or narrow the test group.
2. Revoke or disable the affected model key, connector, or reviewer path when credentials or authority are involved.
3. Preserve request, workflow, and decision records for investigation.
4. Use Instant Rollback or vercel rollback to route production traffic to the previous known-good deployment.
5. Verify the affected path and inspect outstanding Workflow SDK runs. Record each run's status and recovery action, then confirm the available SDK or dashboard operation before attempting recovery.
6. Fix the cause in a preview, then promote the verified deployment to restore normal production assignment.

A deployment rollback does not reverse database writes, schema migrations, external API actions, changed environment variables, or human decisions. Name a separate recovery action and owner for each stateful side effect. Prefer backward-compatible migrations so the previous deployment can still read the current schema.

Run an incident exercise: assume a deployment changes the routing policy and restricted-data requests no longer reach Security. Name the first action that stops incorrect routing, the deployment you would restore, the records you would inspect, and the person authorized to resume processing.

Connect evidence to a decision

Return to the baseline and target in docs/readiness.md. Set a review date and write the decision rule in docs/operating-report.md before reading the final numbers:

Broaden the rollout when:
Continue the small test when:
Retire when:
Decision owner:

Include unresolved access rules, failed evaluations, and support coverage in the decision rule. Frequent use alone is not enough to broaden access if requests are routed incorrectly or nobody can respond to a failure.

Commit the report:

git add docs/operating-report.md
git commit -m "docs: record operating evidence"

Summary

The operating report combines application records, errors, evaluation results, and AI spend. Use it to compare Vendor Review with the original target and decide what to test next. Keep a recovery plan for stored data and running workflows as well as the deployed code.

Check your work

Choose one assessment failure. Starting from the warning, locate the request record, workflow run, model and assessment version, and owner. Then name what you would stop, which deployment you would restore, and which database or workflow state needs separate recovery.