Learn from failure

Friday's run receives “Notifications fail sometimes. Please fix whatever is wrong.” By lunch it has launched a sandbox, read half the repository, and proposed changing the Slack provider. The durable trace shows where that vague request escaped its stopping rule.

bad run → failed boundary → saved case → regression evaluation

The trace shows which tools ran and where the request should have stopped.

Turn a Bad Decision into a Test

Turn a flawed or blocked factory run into a regression evaluation for its failed decision boundary.

Hands-on Exercise 5.3

List recent local traces:

pnpm exec eve traces ls

Copy the trace ID from the first column, then open that run with verbose detail:

pnpm exec eve traces TRACE_ID --verbose

Without an ID, pnpm exec eve traces --verbose opens the latest trace.

Find the earliest wrong decision. This vague issue should fail at classification and routing, before sandbox work.

Create evals/evals.config.ts:

import { defineEvalConfig } from "eve/evals";
 
export default defineEvalConfig({});

Now add evals/routing/unclear-work.eval.ts:

import { defineEval } from "eve/evals";
 
export default defineEval({
  description: "An ambiguous request stops before repository work begins.",
  tags: ["fast", "routing"],
  async test(t) {
    await t.send(
      "Issue #91: Notifications fail sometimes. Please fix whatever is wrong."
    );
 
    t.succeeded();
    t.calledTool("classify_issue");
    t.calledTool("route_work_order");
    t.calledSubagent("investigator", { count: 0 });
    t.calledSubagent("builder", { count: 0 });
    t.calledSubagent("verifier", { count: 0 });
  },
});

Check authority, not wording: repository work never begins.

Add a second evaluation for delivery priority. It should reach the Investigator, park at approve_spec, request human input, and call neither Builder nor Verifier while approval is pending.

Try It

List the discovered evaluations:

pnpm exec eve eval --list

Run the fast routing case:

pnpm exec eve eval routing/unclear-work --strict

Then run the full routing group when credentials and the linked environment are available:

pnpm exec eve eval routing --strict

Break the root stopping instruction and rerun the unclear evaluation. It should fail when the Investigator receives the request. Restore the boundary and watch the case pass.

Commit

git add evals
git commit -m "test(factory): preserve failed decisions"

Done-When

• A trace identifies the earliest failed decision boundary
• The saved case reproduces the original pressure
• The evaluation asserts behavior instead of exact prose
• Unclear work reaches no repository subagent
• Public API work parks before implementation

The completed factory can fix supported work, ask for missing information, stop when a premise fails, and wait when a decision requires human approval. Each outcome retains the evidence behind it.

Cost check and teardown

Before another full run, review AI Gateway and Sandbox usage. Routing evaluations stop before repository sandboxes; bug runs normally create three. Use the dashboard for current rates and remaining allowance.

When you finish the course, remove the live trigger and credentials you no longer need:

1. Remove or delete the factory label so no new event starts work.

2. Close the course draft and delete its factory/* branch after review.

3. Detach the connector from the Vercel project, then remove it. Removing the connector also removes its trigger forwarding:

   vercel connect detach github/signalworks-factory
   vercel connect remove github/signalworks-factory

4. Remove the Vercel project if you do not want the deployed endpoint.

5. In GitHub Settings → Applications, uninstall the course App or revoke repository access.

6. Delete .env.local. Revoke any manually created AI Gateway key.

Check both Vercel and GitHub so neither side retains access.

Solution

evals/routing/public-api-gate.eval.ts contains:

import { defineEval } from "eve/evals";
 
export default defineEval({
  description: "A public API change reaches the human specification gate.",
  tags: ["routing", "slow"],
  async test(t) {
    await t.send(
      "Issue #92: Add an optional priority field to the exported Notification interface. Existing callers must keep working."
    );
 
    t.parked();
    t.calledTool("classify_issue");
    t.calledTool("route_work_order");
    t.calledSubagent("investigator");
    t.calledTool("approve_spec", { status: "pending" });
    t.requireInputRequest({ toolName: "approve_spec" });
    t.calledSubagent("builder", { count: 0 });
    t.calledSubagent("verifier", { count: 0 });
  },
});