Start a software factory with recurring work that has a clear endpoint and evidence a reviewer can check. Issue investigation is a useful candidate when code changes would be premature. Reproducible bug fixes or documentation corrections can fit when you can define acceptance before the agent begins.
Choose the workload before following an implementation guide such as Vercel's guide to building a software factory with eve. A deployed template still needs a task that your team knows how to evaluate.
Copy link to headingWhat makes a task a good first candidate?
Look for a repeated decision your team already understands. Can a maintainer describe an acceptable result without watching the agent work? Can someone inspect that result in less time than it takes to redo the task?
Then examine access. An investigation that reads public issue history needs different permissions from a workflow that changes application data. Prefer a first workload whose required access you can describe and constrain.
Frequency matters, but volume alone isn't a reason to automate. A queue of vague feature requests may contain more work than a queue of reproducible defects while providing a worse starting point. The pilot needs enough consistency to reveal whether the process works.
The following table proposes candidates. A task's actual risk depends on the repository and the consequence of a wrong result.
Avoid classifying every documentation change as low risk. An incorrect authentication example can have more serious consequences than a cosmetic application bug. Review the content of the task, not only its label.
Copy link to headingCan investigation be valuable without generating code?
The Next.js team's closability agent investigated reports after inactivity-based closure had proved unreliable. It searched repository history and, when needed, attempted reproduction across releases. It also sought evidence against its initial conclusion. The agent could not close issues or push changes; maintainers acted on its findings.
The September 2026 post reports 1,462 closures across the repository, including closures outside this review. Of those, 543 concerned already-fixed issues and 278 were duplicates. Those outcomes explain why investigation deserved its own workflow. The headline is not a count of new bug fixes authored by the agent.
For your own pilot, ask the agent to separate observations from its recommendation. A report might reference a merged fix, but the investigation still needs to establish whether the fix addresses the reported behavior. A failed reproduction should name the version and conditions attempted.
Give reviewers a way to disagree. Record whether they accepted the recommendation, corrected it, or needed more evidence. Those outcomes tell you which parts of the investigation deserve attention before you automate an action based on it.
Copy link to headingHow would you choose between investigation and implementation?
Imagine your team has two recurring queues. One contains reports that may concern versions you no longer support. The other contains current regressions with reliable reproductions. This is an illustrative selection exercise, not a result from the Next.js project.
For the first queue, generating a patch may be wasted effort. The decision you need is whether the problem still affects a supported release. Give an investigation workflow that question and require enough evidence for a maintainer to resolve the report.
For the second queue, the team already knows what is broken. A fix workflow may be the better pilot if it can run the reproduction and demonstrate the expected behavior after the change. Its output should include the proposed patch and the checks needed for acceptance.
If both queues are large, use the review burden to break the tie. Sample the work manually and record where maintainers spend their time. Choose one queue for the initial pilot, with an explicit rule for tasks that belong in the other. That keeps an uncertain investigation from silently becoming an implementation project.
Copy link to headingHow should you scope the pilot?
Write a brief that a second maintainer can use to evaluate the work. This example is a proposed starting format for an issue-investigation pilot.
Before the first run, define the evidence a reviewer will need. Keep task selection and acceptance criteria stable during the initial comparison, and record any exceptions.
Use a small historical batch where maintainers already understand the outcome, then try new work. Include at least one ambiguous report so you can see whether the system stops appropriately. Historical results need care: make sure the agent cannot read the eventual resolution if you are testing whether it can investigate the original problem.
Set the batch size according to review capacity. Every result in the first batch should get attention. A hundred unattended proposals won't help you learn which ten were sound.
Copy link to headingWhen should you postpone a legacy rewrite?
A rewrite adds the problem of discovering what the old system does. Hidden behavior can turn an apparently bounded change into a sequence of new requirements.
LaunchDarkly's account of rewriting its targeting interface describes a missed six-week target and a shift toward phased work with human steering. At publication, the new frontend was in internal use, with customer rollout still ahead.
The team had generated about 36,000 lines by the end of week six without finishing the rewrite. As missing behavior emerged, the planned 22 phases became 34. The report makes a useful distinction between producing an implementation and discovering the full scope of a replacement.
If your team cannot state the behavior that must remain equivalent, make that investigation a separate task. Pick a bounded user flow, record the current behavior, and have the owner resolve inconsistencies before asking an agent to replace it.
Copy link to headingWhat shouldn't the first pilot promise?
The pilot should answer whether the selected workflow produces useful results at an acceptable review cost. It cannot establish that the same system handles all repositories or task types.
Flaky checks need special attention. Record known failures and require the agent to report them. Don't allow the workflow to treat repeated retries as evidence that a change is correct.
Expand only after the team understands common failures and can operate the existing handoff. A sensible next step may be a better investigation report rather than permission to make more changes.
Copy link to headingFrequently asked questions
Copy link to headingCan the first workflow be research-only?
Yes. A pilot can prepare evidence for a maintainer without modifying the repository or issue tracker. Evaluate whether its findings reduce the work needed to make a sound decision.
Copy link to headingShould I start with a full legacy rewrite?
Usually, choose a smaller unit whose expected behavior you can describe first. An existing implementation helps you investigate requirements, but it doesn't remove the need to identify edge cases and decide which behavior to preserve.
Copy link to headingCan I run a pilot if tests are flaky?
A research workflow may still be useful while you improve the checks. For code changes, record known instability and require a reviewer to resolve failures that the available evidence cannot explain.
Copy link to headingIs there a standard number of tasks for a pilot?
No. Choose a batch your team can review in full and that includes the kinds of uncertainty the workflow will encounter. Treat a small sample as an initial diagnostic, with more observation needed before broad expansion.