Measure software factory performance by the work your team accepts, the effort required to accept it, and the quality that holds afterward. Track cost per accepted task alongside review time and rework. Count failed attempts in the cost, and compare tasks with similar scope before deciding that performance improved.
You can apply this measurement approach to an AI software factory built as a custom system or from a template such as Vercel's software factory built on eve. The scorecard below is a proposal for operators, not a list of built-in product analytics.
Copy link to headingWhat counts as a completed task?
Define completion at the point where the workflow produces an outcome the team accepts. For an investigation, that could be a finding a maintainer uses to make a decision. For implementation, it could be a merged change. If your goal is successful release, continue tracking through deployment and the relevant production check.
Don't combine these outcomes into an unlabeled count. Ten accepted investigations and ten merged bug fixes answer different operational questions. Keep them in separate groups even when the same agent system handles both.
Retain a task identifier across retries and follow-up runs. A task that needs four attempts remains one task. Otherwise, changes in retry behavior will change both your cost figures and your apparent output.
Copy link to headingWhich metrics belong in the first scorecard?
Start with measures you can trace to work records. Define the observation period before comparing results, especially when later defects can change your assessment of an accepted task.
Report counts next to rates. A result based on three accepted tasks deserves a different interpretation from one observed over hundreds. Keep pending tasks visible so a batch with unfinished work doesn't look like a completed comparison.
Uber's software factory efficiency account separates spending drivers such as session frequency and tokens per request. For managed agents, it also tracks outcome costs and quality signals. Its uReview benchmark uses real pull requests with known bugs, measuring detection quality alongside cost and operational behavior such as timeouts.
That approach suggests two questions for your scorecard: what caused the bill to change, and did the accepted work improve? A lower bill after fewer tasks entered the system answers the first question. It provides no evidence that an individual task became cheaper to complete.
The scorecard definitions here are proposed for this article. Uber's measurements provide context, without supplying a universal performance target.
Copy link to headingHow do you calculate cost per accepted task?
Include all attempts for the selected group. If a workflow spends money on proposals that the team rejects, those costs still belong to operating the workflow.
Here is a hypothetical calculation using dollars in one consistent billing currency. These numbers are illustrative and are not Vercel prices or measured factory results.
The batch has a 60% acceptance rate. Direct cost is $20 per accepted task, calculated as $240 divided by 12. Human effort is 30 minutes per accepted task, calculated as six hours divided by 12.
Keep human time separate until you have an agreed labor-cost assumption. A fuller accounting should also include the engineering effort to build and maintain the factory. Allocate that effort over an explicit period rather than quietly treating setup as free.
The example establishes unit costs. It doesn't establish that the factory saved money, because it contains no comparison with the previous workflow.
Copy link to headingWhat if a cheaper configuration creates more work for reviewers?
Extend the hypothetical batch with a second configuration. Keep the workload and acceptance criteria comparable, and assume both batches have finished. The table illustrates a comparison you could make; it is not a vendor benchmark.
Configuration B cuts direct spend by $60, but requires four more hours of human effort. A team with limited review capacity may prefer A even before converting those hours to a cost. Neither configuration has demonstrated better post-acceptance quality in this example because that observation is missing.
Inspect the extra review work before deciding what to change. Did B produce incorrect patches, omit evidence, or submit changes that were too large? Those problems call for different adjustments. Lowering the model bill alone doesn't tell you which adjustment will improve the result.
Copy link to headingHow can you compare results fairly?
Group work by task type and relevant difficulty. A month with more documentation corrections can show a lower average cost even if no part of the system improved. Record changes in selection rules so a narrower workload doesn't appear to be a more capable agent.
For a model or prompt change, compare versions using a retained set of representative tasks. Keep the acceptance standard stable. Prevent the evaluation from exposing known answers when those answers would give the agent information unavailable in normal use.
Compare against the work your team would otherwise do. Historical timing can provide context, but it may omit interruptions or failed attempts. Label those gaps. For a prospective pilot, ask reviewers to record their effort while the work happens.
Separate execution time from waiting time. A faster agent may have little effect on time to acceptance if proposals wait several days for review. Look at the distribution of delays as well as the average.
Copy link to headingWhen should you expand or revise the workflow?
Set the team's acceptance criteria for expansion before inspecting the results. Those criteria should reflect the consequence of an incorrect outcome and the review capacity available. There is no universal acceptance-rate target that makes every workload worth automating.
If direct cost falls while rework rises, investigate the errors before increasing volume. If the output is sound but reviewer effort remains high, examine the evidence packet and task size. If most runs stop because requirements are unclear, improve intake before changing the model.
Stop expansion when the team cannot review the resulting queue within its intended service window. Record the constraint so the next improvement addresses the work that is delaying acceptance.
Copy link to headingWhat can't this scorecard tell you?
An accepted task can still be low-value work. The scorecard measures the operation of a selected workflow; it doesn't decide whether the team chose the right product priorities.
Short observation periods also miss later defects and ongoing maintenance. Preserve enough task history to revisit earlier conclusions. State the period covered whenever you share a result.
Copy link to headingFrequently asked questions
Copy link to headingIs the number of generated pull requests a useful success metric?
It is a workload-volume measure, but it doesn't establish useful delivery. Pair it with acceptance and the effort required to review or correct the proposals.
Copy link to headingDoes lower inference spend mean the factory is more efficient?
No. Lower model spend can coincide with additional retries or greater human effort. Compare the total cost of producing accepted results while checking that quality remains acceptable.
Copy link to headingCan I compare documentation tasks with bug fixes?
Track the categories separately before combining them. Changes in the mix of tasks can move an overall average even when performance within each category stays the same.
Copy link to headingHow should I report a small pilot?
Show raw counts, costs, and unresolved work alongside calculated rates. Describe the pilot's scope and observation period so readers can judge how much evidence the results provide.