---
title: "Parse Test Failures"
description: "Test runner output is a wall of text. Useful for humans, useless for merging with structured AI findings. In this lesson, we write a small parser that pulls failure lines out of stdout/stderr and shapes them into typed `TestFinding` records."
canonical_url: "https://vercel.com/academy/vercel-sandbox/parse-test-failures"
md_url: "https://vercel.com/academy/vercel-sandbox/parse-test-failures.md"
docset_id: "vercel-academy"
doc_version: "1.0"
last_updated: "2026-05-18T03:58:27.607Z"
content_type: "lesson"
course: "vercel-sandbox"
course_title: "Vercel Sandbox"
prerequisites:  []
---

<agent-instructions>
Vercel Academy — structured learning, not reference docs.
Lessons are sequenced.
Adapt commands to the human's actual environment (OS, package manager, shell, editor) — detect from project context or ask, don't assume.
The lesson shows one path; if the human's project diverges, adapt concepts to their setup.
Preserve the learning goal over literal steps.
Quizzes are pedagogical — engage, don't spoil.
Quiz answers are included for your reference.
</agent-instructions>

# Parse Test Failures

# Parse Test Output into Structured Findings

Test runner output is a wall of text. Useful for humans reading a terminal, useless for anything else.

We want to put test failures next to AI findings in the same report. That means giving them the same shape: severity, summary, location. A small parser turns "FAIL src/auth.test.ts > login rejects empty password" into a structured record we can sort, count, and serialize.

## Outcome

Create `src/test-runner.ts` with a `parseTestFailures(output)` function that returns an array of `TestFinding` objects, one per failed assertion or suite.

## Fast Track

1. Create `src/test-runner.ts`.
2. Define a `TestFinding` type with `severity`, `category`, `summary`, and `details`.
3. Write `parseTestFailures(output)` that filters lines for failure markers and maps them.

## Hands-on exercise

Create `src/test-runner.ts`:

```ts
export type TestFinding = {
  severity: 'medium' | 'high';
  category: 'test-failure';
  summary: string;
  details: string;
};

const FAILURE_MARKERS = ['FAIL ', '✕ ', '× '];

export function parseTestFailures(output: string): TestFinding[] {
  const lines = output.split('\n');

  return lines
    .map((line) => line.trim())
    .filter((line) => FAILURE_MARKERS.some((marker) => line.includes(marker)))
    .map((line) => ({
      severity: 'high' as const,
      category: 'test-failure' as const,
      summary: 'Automated test failure',
      details: line
    }));
}
```

A few things worth pointing out.

The parser is intentionally dumb. It looks for known failure markers and pulls those lines out. It doesn't try to attribute failures to specific source files, doesn't try to count assertions, doesn't try to extract diff output. Test runners are wildly inconsistent in how they format failures, and a generic parser that gets 80% of cases right is far more useful than a fragile one that breaks on the next runner.

The markers cover the most common shapes:

- `FAIL ` is what vitest, jest, and most node runners prefix failed suites with
- `✕ ` is the per-test failure character vitest uses
- `× ` is what mocha and a few others use

If a runner uses a different marker, the parser silently misses those failures. That's a known limitation, not a bug. In a real tool you'd extend this list as you encounter new runners.

We're also typing severity as `'high'` for every failure. A failing test is a failing test; the parser isn't smart enough to distinguish "this one assertion is wrong" from "the whole suite blew up." The AI findings keep the four-level severity scale; test findings are binary.

\*\*Warning: Troubleshooting: no failures detected when tests are red\*\*

If `pnpm test` exits non-zero but `parseTestFailures` returns an empty array, the runner is using a marker we don't recognize. Print the raw output, find the failure line, and add its prefix to `FAILURE_MARKERS`.

\*\*Note: Troubleshooting: too many false positives\*\*

If a passing test run produces "findings" anyway, you're probably matching against output that contains the word "FAIL" but isn't a real failure (a test *for* failure behavior, for instance). Tighten the marker to `^FAIL ` (anchored start) if needed.

## Try It

Test the parser without booting a Sandbox. Add a temporary check at the bottom of `src/test-runner.ts`:

```ts
const sampleOutput = `
> repo@1.0.0 test
> vitest run

  ✓ src/sum.test.ts (3)
  ✕ src/auth.test.ts > login rejects empty password
  ✕ src/auth.test.ts > login rejects short password

  Test Files  1 failed | 1 passed
  Tests       2 failed | 3 passed
  FAIL  src/auth.test.ts
`;

console.log(parseTestFailures(sampleOutput));
```

Run it:

```bash
pnpm tsx src/test-runner.ts
```

Expected output:

```js
[
  {
    severity: 'high',
    category: 'test-failure',
    summary: 'Automated test failure',
    details: '✕ src/auth.test.ts > login rejects empty password'
  },
  {
    severity: 'high',
    category: 'test-failure',
    summary: 'Automated test failure',
    details: '✕ src/auth.test.ts > login rejects short password'
  },
  {
    severity: 'high',
    category: 'test-failure',
    summary: 'Automated test failure',
    details: 'FAIL  src/auth.test.ts'
  }
]
```

Three structured records. Delete the sample block before moving on.

## Commit

```bash
git add src/test-runner.ts
git commit -m "feat(testing): parse runner output into structured findings"
```

## Done-When

- [ ] `src/test-runner.ts` exports `TestFinding` type and `parseTestFailures` function
- [ ] Sample output produces one finding per failure line
- [ ] Passing-only output produces an empty array
- [ ] Each finding has `severity: 'high'` and `category: 'test-failure'`

## Solution

```ts title="src/test-runner.ts"
export type TestFinding = {
  severity: 'medium' | 'high';
  category: 'test-failure';
  summary: string;
  details: string;
};

const FAILURE_MARKERS = ['FAIL ', '✕ ', '× '];

export function parseTestFailures(output: string): TestFinding[] {
  const lines = output.split('\n');

  return lines
    .map((line) => line.trim())
    .filter((line) => FAILURE_MARKERS.some((marker) => line.includes(marker)))
    .map((line) => ({
      severity: 'high' as const,
      category: 'test-failure' as const,
      summary: 'Automated test failure',
      details: line
    }));
}
```


---

[Full course index](/academy/llms.txt) · [Sitemap](/academy/sitemap.md)