Run the Test Suite in the Sandbox

Static analysis is helpful, but if tests are red and we ignore them, that is code review cosplay.

The whole reason Sandboxes exist is so we can actually execute the code, not just read it. The model is allowed to be wrong, and the tests are the ground truth for at least one definition of "works." So let's run them.

Outcome

Extend the lifecycle to run pnpm test inside the Sandbox after the file collection step, and return the test command's exit code, stdout, and stderr.

Fast Track

After file collection, run cd repo && pnpm install so dependencies are present.
Run cd repo && pnpm test and capture the result.
Return testResult: { exitCode, stdout, stderr } from the lifecycle.

Hands-on exercise

Open src/sandbox-lifecycle.ts and add the test step:

import { Sandbox } from '@vercel/sandbox';
 
const INTERESTING_PATHS = [
  'repo/package.json',
  'repo/src/index.ts',
  'repo/src/app.ts',
  'repo/lib/auth.ts'
];
 
export type TestResult = {
  exitCode: number;
  stdout: string;
  stderr: string;
};
 
export type LifecycleResult = {
  sandboxId: string;
  cloneExitCode: number;
  files: Array<{ path: string; content: string }>;
  testResult: TestResult;
};
 
export async function runSandboxLifecycle(repoUrl: string): Promise<LifecycleResult> {
  const sandbox = await Sandbox.create();
 
  try {
    const clone = await sandbox.runCommand(`git clone ${repoUrl} repo`);
    if (clone.exitCode !== 0) {
      throw new Error(`Clone failed: ${clone.stderr}`);
    }
 
    const files: Array<{ path: string; content: string }> = [];
    for (const fullPath of INTERESTING_PATHS) {
      try {
        const content = await sandbox.readFile(fullPath);
        files.push({
          path: fullPath.replace(/^repo\//, ''),
          content
        });
      } catch {
        // file doesn't exist in this repo; skip it
      }
    }
 
    await sandbox.runCommand('cd repo && pnpm install');
    const test = await sandbox.runCommand('cd repo && pnpm test');
 
    return {
      sandboxId: sandbox.sandboxId,
      cloneExitCode: clone.exitCode,
      files,
      testResult: {
        exitCode: test.exitCode,
        stdout: test.stdout,
        stderr: test.stderr
      }
    };
  } finally {
    await sandbox.stop();
  }
}

A couple of intentional choices.

We're not throwing when pnpm test exits non-zero. Failing tests are a finding, not a crash. The whole point of running them is that the failures become part of the review.

We're also not parsing anything yet. Parsing is the next lesson. Right now we just need to prove the tests ran and we captured the output.

Troubleshooting: pnpm is not available

Some Sandbox base images don't have pnpm. If pnpm install returns "command not found", we'll handle that in 4.3 when we detect the lockfile and pick the matching tool. For now, pick a repo that uses pnpm.

Troubleshooting: install takes forever

pnpm install on a real repo can take 30-90 seconds. That's normal for a cold Sandbox. We'll speed it up with snapshots in Chapter 5.

Try It

Pick a repo that uses pnpm and has a test script (a small one — install time matters):

pnpm review https://github.com/<a-pnpm-repo-with-tests>

Expected output (the trailing test summary varies):

Reviewing https://github.com/<...>...
Sandbox: sbx_7N2k4A...
Collected 2 file(s) for analysis.
Overall risk: low
Findings: 3
  [medium] ...
  ...

You won't see the test output in the terminal yet because the CLI isn't logging it. To verify it's there, temporarily add console.log(lifecycle.testResult) in the CLI right after the lifecycle call. You should see something like:

{
  exitCode: 0,
  stdout: '> repo@1.0.0 test\n> vitest run\n\n  ✓ src/sum.test.ts (3)\n...',
  stderr: ''
}

Or if tests failed:

{
  exitCode: 1,
  stdout: '... FAIL  src/sum.test.ts > adds correctly ...',
  stderr: ''
}

Either way, the suite ran and we captured the result. Remove the debug log before committing.

Commit

git add src/sandbox-lifecycle.ts
git commit -m "feat(sandbox): run repo test suite and capture the result"

Done-When

pnpm install runs in the Sandbox before tests
pnpm test runs after install
testResult: { exitCode, stdout, stderr } is returned from the lifecycle
Non-zero test exit codes don't crash the lifecycle

Solution

src/sandbox-lifecycle.ts

import { Sandbox } from '@vercel/sandbox';
 
const INTERESTING_PATHS = [
  'repo/package.json',
  'repo/src/index.ts',
  'repo/src/app.ts',
  'repo/lib/auth.ts'
];
 
export type TestResult = {
  exitCode: number;
  stdout: string;
  stderr: string;
};
 
export type LifecycleResult = {
  sandboxId: string;
  cloneExitCode: number;
  files: Array<{ path: string; content: string }>;
  testResult: TestResult;
};
 
export async function runSandboxLifecycle(repoUrl: string): Promise<LifecycleResult> {
  const sandbox = await Sandbox.create();
 
  try {
    const clone = await sandbox.runCommand(`git clone ${repoUrl} repo`);
    if (clone.exitCode !== 0) {
      throw new Error(`Clone failed: ${clone.stderr}`);
    }
 
    const files: Array<{ path: string; content: string }> = [];
    for (const fullPath of INTERESTING_PATHS) {
      try {
        const content = await sandbox.readFile(fullPath);
        files.push({
          path: fullPath.replace(/^repo\//, ''),
          content
        });
      } catch {
        // file doesn't exist in this repo; skip it
      }
    }
 
    await sandbox.runCommand('cd repo && pnpm install');
    const test = await sandbox.runCommand('cd repo && pnpm test');
 
    return {
      sandboxId: sandbox.sandboxId,
      cloneExitCode: clone.exitCode,
      files,
      testResult: {
        exitCode: test.exitCode,
        stdout: test.stdout,
        stderr: test.stderr
      }
    };
  } finally {
    await sandbox.stop();
  }
}