01 / FIELD NOTES
A cheaper attempt can cost more to finish
This is a worked calculation with hypothetical data, not a measured model benchmark. Its purpose is to make the evaluation method visible so you can replace the numbers with results from your own workload.
Assume two systems each receive 1,000 attempts drawn from the same task distribution. System A costs $0.10 an attempt and produces 900 accepted results. System B costs $0.04 an attempt and produces 300 accepted results. With no repair labor, A spends $100 and averages $0.111 per accepted result. B spends $40 and averages $0.133. Lower spending bought far less completed work.
02 / FIELD NOTES
Normalize the amount of completed work
The first comparison reports what happened to equal attempt counts. A purchase decision also needs the cost of delivering a fixed volume of accepted results. Use replayable, representative tasks and run the complete workflow, including retries and fallback, rather than multiplying an average acceptance probability without testing it.
Failures can be correlated. A model that misunderstands a document may fail again on the same document. A retry policy is part of the evaluated system. Comparing the best run from one model with the first run from another produces a misleading result.
- Choose acceptance criteria before seeing model outputs.
- Run the same task set and workflow constraints for each system.
- Keep failed attempts in the cost denominator’s supporting ledger.
- Report accepted count, rejection reasons, retries, latency, and total spend together.
03 / FIELD NOTES
Add the cost of repair
Extend the example: A needs 30 minutes of human correction and B needs 120 minutes, both valued at an illustrative $60 per hour. A’s delivery cost becomes $130, or $0.144 per accepted result. B’s becomes $160, or $0.533. These figures assume that the reported accepted counts already include the work repaired during that period; they are not a claim about any named model.
Separate active repair time from time a person could use for something else. Value recovered time conservatively. Avoid counting review both as a flat operating allowance and as per-task labor. The same cost should enter the ledger once.
04 / FIELD NOTES
Use a task ledger you can audit
For each task, retain a task ID, model and configuration, total billed input and output, tool charges, attempts, wall time, reviewer decision, rejection reason, and active repair minutes. Use a representative sample and report its size and selection method.
In the infrastructure calculator, enter total monthly API spending inclusive of failed attempts, the number of accepted tasks, and the operating labor required by the proposed replacement. The resulting unit cost is meaningful only if all options can serve the same accepted workload at the required service level.
05 / FIELD NOTES
Keep business ROI as a second comparison
A lower delivery cost is evidence about the workflow. Business ROI also depends on adoption, integration, maintenance, quality, and what the accepted result allows the organization to do. Compare the process with AI against the process without it.
Publish your actual task set, acceptance rules, cost ledger, and limitations before making a broad model recommendation. A small controlled study can be useful without pretending to establish a universal winner.
Sources & next steps
Published November 2, 2026. Worked examples are hypothetical; benchmark methods describe measurements to perform.
MAKE THE ECONOMICS VISIBLE