---
title: "Learn From Failure"
description: "Inspect durable traces, locate the decision boundary that failed, and encode the case as an eve evaluation that protects the factory from repeating the same mistake."
canonical_url: "https://vercel.com/academy/creating-a-software-factory/learn-from-failure"
md_url: "https://vercel.com/academy/creating-a-software-factory/learn-from-failure.md"
docset_id: "vercel-academy"
doc_version: "1.0"
last_updated: "2026-08-29T01:48:00.808Z"
content_type: "lesson"
course: "creating-a-software-factory"
course_title: "Creating a Software Factory"
prerequisites:  []
---

<agent-instructions>
Vercel Academy — structured learning, not reference docs.
Lessons are sequenced.
Adapt commands to the human's actual environment (OS, package manager, shell, editor) — detect from project context or ask, don't assume.
The lesson shows one path; if the human's project diverges, adapt concepts to their setup.
Preserve the learning goal over literal steps.
Quizzes are pedagogical — engage, don't spoil.
Quiz answers are included for your reference.
</agent-instructions>

# Learn From Failure

# Learn from failure

Friday's run receives “Notifications fail sometimes. Please fix whatever is wrong.” By lunch it has launched a sandbox, read half the repository, and proposed changing the Slack provider. The durable trace shows where that vague request escaped its stopping rule.

```text
bad run → failed boundary → saved case → regression evaluation
```

The trace shows which tools ran and where the request should have stopped.

## Turn a Bad Decision into a Test

Turn a flawed or blocked factory run into a regression evaluation for its failed decision boundary.

## Hands-on Exercise 5.3

List recent local traces:

```bash
pnpm exec eve traces ls
```

Copy the trace ID from the first column, then open that run with verbose detail:

```bash
pnpm exec eve traces TRACE_ID --verbose
```

Without an ID, `pnpm exec eve traces --verbose` opens the latest trace.

Find the earliest wrong decision. This vague issue should fail at classification and routing, before sandbox work.

Create `evals/evals.config.ts`:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";

export default defineEvalConfig({});
```

Now add `evals/routing/unclear-work.eval.ts`:

```ts title="evals/routing/unclear-work.eval.ts"
import { defineEval } from "eve/evals";

export default defineEval({
  description: "An ambiguous request stops before repository work begins.",
  tags: ["fast", "routing"],
  async test(t) {
    await t.send(
      "Issue #91: Notifications fail sometimes. Please fix whatever is wrong."
    );

    t.succeeded();
    t.calledTool("classify_issue");
    t.calledTool("route_work_order");
    t.calledSubagent("investigator", { count: 0 });
    t.calledSubagent("builder", { count: 0 });
    t.calledSubagent("verifier", { count: 0 });
  },
});
```

Check authority, not wording: repository work never begins.

Add a second evaluation for delivery priority. It should reach the Investigator, park at `approve_spec`, request human input, and call neither Builder nor Verifier while approval is pending.

## Try It

List the discovered evaluations:

```bash
pnpm exec eve eval --list
```

Run the fast routing case:

```bash
pnpm exec eve eval routing/unclear-work --strict
```

Then run the full routing group when credentials and the linked environment are available:

```bash
pnpm exec eve eval routing --strict
```

Break the root stopping instruction and rerun the unclear evaluation. It should fail when the Investigator receives the request. Restore the boundary and watch the case pass.

\*\*Note: Choose the earliest assertion\*\*

For any flawed run, identify the first tool or subagent call that should differ. An early behavioral assertion produces a faster and more useful evaluation.

\*\*Warning: The evaluation grades prose\*\*

Assert the route, tool calls, approval state, and absent side effects. Exact response wording makes a brittle test of style.

\*\*Warning: The evaluation repeats a real side effect\*\*

Use a case that stops or parks before branch and pull request creation. Keep destructive or costly integration cases isolated and deliberate.

## Commit

```bash
git add evals
git commit -m "test(factory): preserve failed decisions"
```

## Done-When

- [ ] A trace identifies the earliest failed decision boundary
- [ ] The saved case reproduces the original pressure
- [ ] The evaluation asserts behavior instead of exact prose
- [ ] Unclear work reaches no repository subagent
- [ ] Public API work parks before implementation

The completed factory can fix supported work, ask for missing information, stop when a premise fails, and wait when a decision requires human approval. Each outcome retains the evidence behind it.

## Cost check and teardown

Before another full run, review AI Gateway and Sandbox usage. Routing evaluations stop before repository sandboxes; bug runs normally create three. Use the dashboard for current rates and remaining allowance.

When you finish the course, remove the live trigger and credentials you no longer need:

1. Remove or delete the `factory` label so no new event starts work.

2. Close the course draft and delete its `factory/*` branch after review.

3. Detach the connector from the Vercel project, then remove it. Removing the connector also removes its trigger forwarding:

   ```bash
   vercel connect detach github/signalworks-factory
   vercel connect remove github/signalworks-factory
   ```

4. Remove the Vercel project if you do not want the deployed endpoint.

5. In GitHub **Settings → Applications**, uninstall the course App or revoke repository access.

6. Delete `.env.local`. Revoke any manually created AI Gateway key.

Check both Vercel and GitHub so neither side retains access.

## Solution

`evals/routing/public-api-gate.eval.ts` contains:

```ts title="evals/routing/public-api-gate.eval.ts"
import { defineEval } from "eve/evals";

export default defineEval({
  description: "A public API change reaches the human specification gate.",
  tags: ["routing", "slow"],
  async test(t) {
    await t.send(
      "Issue #92: Add an optional priority field to the exported Notification interface. Existing callers must keep working."
    );

    t.parked();
    t.calledTool("classify_issue");
    t.calledTool("route_work_order");
    t.calledSubagent("investigator");
    t.calledTool("approve_spec", { status: "pending" });
    t.requireInputRequest({ toolName: "approve_spec" });
    t.calledSubagent("builder", { count: 0 });
    t.calledSubagent("verifier", { count: 0 });
  },
});
```


---

[Full course index](/academy/llms.txt) · [Sitemap](/academy/sitemap.md)