Measure adoption, quality, and cost

Request volume alone cannot show that Vendor Review improves the review process. Return to the baseline and target in docs/readiness.md.

Use the traffic and test requests available during the course, and label that short window honestly. The goal is to prove the measurement path and set the next reporting period, not manufacture an adoption claim from sample activity.

Adoption

Use Observability to inspect traffic, function errors, and external calls. Some route-level latency and path breakdowns require Observability Plus, so record which signals your plan exposes rather than treating every dashboard view as universally available. Measure submitted requests, active users or teams where identity permits, completion rate, and time from submission to decision from the application’s persisted records. Separate page views from completed workflows.

Reliability

Track HTTP errors, assessment failures, workflow failures, decision conflicts, and latency. Inspect one request across application logs and its workflow run. Record how an operator moves from an alert to the affected decision record.

If your organization has access to the Security Dashboard, inspect its organization-wide findings alongside these project-level checks. The Security Dashboard is in Private Beta and surfaces security posture across accounts and projects, including missing MFA, shared secrets, and long-lived credentials. Record it as demonstrated or planned unless you actually inspected your organization there.

AI quality

Rerun the evaluation set from Make AI Decisions Dependable. Track deterministic-policy test failures separately from model-assessment failures. Model or prompt changes should show their evaluation result beside the deployed assessment version.

Cost

Use AI Gateway usage to attribute requests, tokens, and cost by model, project, and credential. Estimate cost per completed vendor request, not just cost per model call. Add a team, project, or API-key budget with enough headroom for expected use and document what happens when it is reached.

Build the first operating report

Add docs/operating-report.md:

Reporting period:
Requests submitted:
Requests completed:
Median and p90 decision time:
Assessment failures:
Workflow failures:
Evaluation pass rate:
AI cost per completed request:
Budget and owner:
Known gaps and who will respond:

Use your actual small-course numbers and label them accordingly. Do not manufacture a convincing dashboard.

Contain and roll back a harmful change

Containment stops additional harm; rollback restores a known-good application version. Add this sequence under Recovery in docs/operating-report.md:

1. Stop new submissions or narrow the test group.
2. Revoke or disable the affected model key, connector, or reviewer path when credentials or authority are involved.
3. Preserve request, workflow, and decision records for investigation.
4. Use Instant Rollback or vercel rollback to route production traffic to the previous known-good deployment.
5. Verify the affected path, inspect outstanding Workflow SDK runs, and decide explicitly whether each should resume, retry, or remain paused.
6. Fix the cause in a preview, then promote the verified deployment to restore normal production assignment.

A deployment rollback does not reverse database writes, schema migrations, external API actions, changed environment variables, or human decisions. Name a separate recovery action and owner for each stateful side effect. Prefer backward-compatible migrations so the previous deployment can still read the current schema.

Run an incident exercise: assume a new assessment version routes restricted-data requests incorrectly. Name the first action that stops incorrect routing, the deployment you would restore, the records you would inspect, and the person authorized to resume processing.

Connect evidence to a decision

Return to the baseline and target in docs/readiness.md. Set a review date and write the decision rule in docs/operating-report.md before reading the final numbers:

Broaden the rollout when:
Continue the small test when:
Retire when:
Decision owner:

Adoption does not automatically justify a broader rollout. An application with frequent use may still have an unresolved access rule, unreliable model behavior, or an owner who cannot support it. The final decision combines the business outcome with the checks completed throughout the course.

Commit the report:

git add docs/operating-report.md
git commit -m "docs: record operating evidence"

Summary

Observability shows whether the application is available. Evaluation shows whether its AI behavior remains useful. AI Gateway shows usage and spend. A production owner needs all three views to decide whether the application is delivering value responsibly.

Check your work

Choose one assessment failure. Starting from the warning, locate the request record, workflow run, model and assessment version, and owner. Then say what you would stop, what deployment you would restore, and what database or workflow state still needs separate recovery. If “roll back” is the entire answer, the stateful half is missing.