Verify software factory code by checking the proposed revision against explicit acceptance criteria. Require evidence of the reported behavior, the effect of the patch, and relevant regression checks. Have a reviewer examine the change independently, and keep unresolved checks visible before the person with merge authority makes a decision.
For a concrete implementation, Foreman's documented review process uses a separate reviewer to assess the pushed branch against acceptance criteria. It returns a draft pull request for human acceptance. The verification practices below are recommendations you can adapt to your own workflow.
Copy link to headingWhat evidence should accompany a proposed change?
A reviewer needs to connect each important requirement to an observation. A statement that tests passed leaves several questions unanswered. Which tests ran? Against which revision? Did those tests exercise the behavior in the request?
Ask the workflow to retain a compact review packet. Keep full logs available by reference so the reviewer can investigate without reading every command on every task.
Give each check a status such as passed, failed, blocked, or not run. Treat blocked and not run as distinct from a failure: the workflow lacks evidence in those cases. The acceptance decision must account for that gap.
Copy link to headingHow do you check that a test proves the right thing?
For a bug fix, try the regression test against the original revision and the proposed revision. Inspect why it fails before the change. A test that fails because its setup is broken doesn't demonstrate the reported defect.
Then compare the assertions with the requirement. Suppose a report says that a filtered export loses rows. A test that checks only whether the export file exists can pass while the bug remains. Verify the exported content using a fixture with known expected rows.
Keep the fixture independent of the new implementation where practical. If the test computes its expected result using the same faulty logic as the code under test, agreement between them gives little confidence.
Review changes to existing tests as well. Sometimes the requested behavior requires new expectations. Sometimes a patch makes a failing check disappear by weakening it. The diff should make that distinction visible.
Copy link to headingWhat does evidence look like in an actual factory change?
The AI SDK factory's blocked-domain feature example connects the request to several observations. An analysis probe established that support was missing. The implementation agent then exercised OpenAI web search with wikipedia.org blocked. Automated review assessed the patch, and Lars Grammel reviewed the evidence and merged PR #18033. Backports received separate review.
For your own workflow, make each observation answer a named requirement. A capability probe establishes a different fact from a runtime test. Keep both when they contribute evidence, and identify the acceptance decision that remains after those checks.
Copy link to headingHow would a reviewer use a complete evidence packet?
Return to the hypothetical filtered-export bug. Assume the requirement is to export every matching row, including results beyond the first page. The packet below illustrates the level of detail a reviewer could request; it is not an AI SDK artifact.
The successful row-count check doesn't resolve the missing permission check. If the patch changes how export queries apply access restrictions, the reviewer should request the fixture or arrange another check before accepting it. Record that decision as a request for evidence, with a named owner.
After another commit arrives, confirm which results remain applicable and rerun affected checks. The evidence packet should describe the revision the team is considering now. Retaining old results is useful for history, but their presence should not make an untested revision appear verified.
Copy link to headingWhat does an independent review add?
Ask the reviewer to begin with the requirements and the proposed diff. The implementer's explanation can be useful context, but it should not be the reviewer's only account of what happened.
Have the reviewer identify missing evidence and examine whether the patch changes more than the task requires. A correct fix can still introduce an unrelated compatibility change that needs a separate decision.
Using another model is one way to vary the review process. It cannot guarantee that the two agents will make different mistakes. The review still needs checks that connect back to observable behavior, plus an escalation path for ambiguity.
Keep approval authority separate from the language in a review response. An automated review that says a patch is acceptable should not gain permission to merge it merely by producing that judgment.
Copy link to headingWhen should you verify behavior in a browser?
Use a browser when acceptance depends on an interaction that repository checks don't adequately exercise. A hypothetical checkout bug might require a specific account state and several steps before it appears. A screenshot of the initial page would not establish that the flow works.
Record the conditions that matter, including the test account's permissions and the state needed to reproduce the problem. Verify the intended preview deployment and retain its association with the proposed revision. Keep sensitive account information out of the review packet.
Vercel's guide to giving a software factory a browser adds KERNEL browser access for attended reproduction and preview verification. It provides the implementation instructions for that extension; browser access should not be assumed to exist in every factory run.
After a browser check, say what you observed. A completed checkout under one account type establishes a narrower result than a claim that checkout works for all users.
Copy link to headingHow should the workflow handle a failed deployment?
Start with the failure evidence. A failed deployment status alone doesn't identify the cause. Preserve the relevant build error and compare it with the proposed changes before beginning another implementation attempt.
The guide to managing Vercel projects with a software factory describes a read-only deployment-context integration scoped to a team and project. That access helps investigation; it does not itself grant deployment authority.
If the failure appears unrelated, record the reason and supporting evidence. Put a limit on repeated attempts, and return to a person when the workflow cannot distinguish an infrastructure problem from a defect in the patch.
Copy link to headingWhat remains outside verification's guarantee?
Even a well-supported review covers specified behavior under particular conditions. It cannot prove the absence of all defects. Match review depth to the consequence of a wrong change, and require an appropriate owner for changes whose acceptance depends on product or architectural judgment.
Preserve the accepted revision and record review time and later rework. A defect found after merge can reveal a missing test or an incomplete requirement. Update the relevant check before sending the same class of work through again.
Copy link to headingFrequently asked questions
Copy link to headingCan an agent write the tests for its own patch?
Yes, but examine whether those tests would detect an incorrect implementation. Check their assertions and, for a regression, verify that the original code fails for the expected reason.
Copy link to headingDoes a second model guarantee a correct review?
No. A separate reviewer can challenge assumptions, but agreement between models is insufficient evidence of correctness. Require observations that demonstrate the requested behavior.
Copy link to headingDoes a successful preview check prove production is safe?
A preview check supports a claim about the conditions exercised in that environment. Production may differ in data or configuration, so define any additional release checks the change needs.
Copy link to headingWhat should happen when the agent cannot verify a requirement?
The workflow should name the missing evidence and return a blocked or incomplete result. A reviewer can then request another check, clarify the requirement, or decide whether the remaining uncertainty is acceptable under the team's policy.