01 / START WITH THE OUTCOME
The cheapest token can produce the most expensive result.
A useful comparison starts with a task someone will accept: a support answer that resolves the issue, an extraction that passes validation, or a code change that survives review. A million generated tokens is activity. It is not necessarily a million tokens of value.
Consider a deliberately simplified example. One system spends $0.10 per attempt and another spends $0.04. If 90% of the first system’s attempts pass your acceptance test, its inference cost per accepted result is about $0.11. If only 30% of the second system’s attempts pass, its cost is about $0.13. The cheaper token lost before you priced the human cleanup.
Cost per accepted result =
total delivery cost ÷ accepted results
That example is arithmetic, not a benchmark or a retry prediction. Real failures are often correlated: repeating the same prompt may simply repeat the same mistake. Measure the entire workflow, including retries, tool calls, review, and repair. Then ask whether the result arrives fast enough to be useful.
This changes the purchasing conversation. Instead of “How much VRAM can we afford?”, start with “What work do we need to complete, at what quality and volume?” Hardware follows the answer.
02 / SEPARATE THE PURCHASES
API, rented GPUs, and local hardware are three operating models.
| Option | Main cost driver | What you take on |
|---|---|---|
| Managed model API | Billable usage and service-specific fees | Integration, evaluation, rate-limit handling, provider dependency |
| Rented GPUs | Allocated GPU-hours, plus operations | Serving, scaling, model updates, reliability |
| Owned hardware | Upfront capital, plus ongoing operations | All of the above, plus physical infrastructure and asset risk |
Open weights describe access to a model. They do not require you to become its infrastructure provider. A managed API can serve an open-weight model; renting a GPU still leaves the serving work with you. Keep model choice and hosting choice separate.
A subscription is not a token budget.
A chat or coding subscription buys access under a product’s plan and usage rules. It is not a promise of a dedicated endpoint or interchangeable API capacity. If your team is comparing a subscription with hardware, first document the workflow and limits you are actually replacing. This calculator uses metered API spending as its baseline.
Memory is a gate, not a business case.
The ideal weight storage of a model is parameter count × bits per parameter ÷ 8. A hypothetical 70-billion-parameter model at 4 bits needs 35 GB for weights alone, using decimal GB. Quantization metadata, runtime buffers, and attention state add to that. Sparse activation in a mixture-of-experts model reduces computation per token; it does not make all the inactive experts disappear from storage.
Getting the weights to load is the first milestone. The next is serving the intended context length and concurrency at acceptable latency. vLLM’s optimization guide describes how cache capacity and serving settings affect that tradeoff. A purchase spreadsheet cannot establish it.
03 / MAKE THE ASSUMPTIONS VISIBLE
The $300,000 question.
Start with the team-server scenario from my original essay: $1,000 a week in API usage, or $52,000 a year; a $300,000 hardware purchase; and $20,000 a year to operate it. These are illustrative assumptions, not a minimum specification or a vendor quote.
If the server eliminates the entire API bill, it saves $32,000 a year after operations. Simple payback is $300,000 ÷ $32,000, or 9.4 years. Dividing the purchase price by the API bill alone gives 5.8 years—and silently assumes free operations.
| Option | Three-year cost |
|---|---|
| API: $52,000 / year | $156,000 |
| Owned: $300,000 + $20,000 / year | $360,000 |
| Rented: 8 × H100 SXM, continuously allocated | $838,858 |
The rental example uses Lambda’s listed $3.99 per GPU-hour for the eight-GPU H100 SXM instance, checked November 2, 2026: 8 × $3.99 × 8,760 hours × 3 years. It is an infrastructure price example, not an equivalent-performance claim or a minimum hardware recommendation. Taxes and rental operating labor are excluded here; add your own costs in the calculator.
To recover the owned server within three years, the setup must eliminate $120,000 in annual API spending: $100,000 toward the purchase and $20,000 toward operations. That means $10,000 a month, provided it can actually serve the workload.
The operator’s time moves the line.
Four hours a week at $100 an hour adds $20,800 a year. Annual savings then fall to $11,200, stretching simple payback to 26.8 years. That is not a hardware lifespan prediction. It is evidence that this scenario fails the purchase test.
Try it above: open “Power & operating labor,” set operating hours to 17.3333 per month and labor to $100 an hour. Then change the workload or hardware price. The conclusion is allowed to change with the evidence.
04 / THE TWO TESTS A SPREADSHEET MISSES
Utilization is not the same as useful demand.
Owning a server converts a variable bill into a commitment. The hardware consumes capital while it is waiting for your next request. Rental changes the financing, but an allocated, idle GPU is still rented.
The tempting response is to keep the machine busy. But additional inference creates savings only when it replaces something you would otherwise pay for—or creates value you can demonstrate. Generating extra output to improve a utilization chart is not an economic win.
The goal is not to maximize GPU utilization.
It is to minimize the cost of work you actually need done.
Average demand is also a poor stand-in for peak capacity. A server that covers most daily tokens can still fail when several people submit long prompts at once. Measure prompt processing, output throughput, time to first token, tail latency, and concurrency on representative requests. Budget for the service level people will experience.
Partial replacement is often the honest answer.
A smaller local model may handle classification, extraction, or other bounded work while harder tasks remain on a remote model. That can be an excellent architecture. It also means the API bill does not disappear.
The calculator’s replacement slider represents the percentage of API dollars eliminated, not the percentage of prompts routed locally. Those can differ dramatically. Moving 80% of inexpensive requests may remove only a small fraction of spending. Both self-hosted options retain the remainder as an API fallback cost.
Validate the replacement rate with a pilot. Include routing overhead, monitoring, model updates, and repair work in operations. Hybrid infrastructure deserves the same scrutiny as a full migration.
05 / A BETTER BUYING PROCESS
Buy evidence before you buy infrastructure.
- Measure a representative week.
Collect input and output volume, the complete bill, accepted tasks, retries, review time, peak concurrency, and latency. Separate a busy week from a typical week.
- Try to lower the managed baseline.
Evaluate a less expensive model, shorter context, caching, or asynchronous batching where the workflow permits it. Use your actual provider terms. A smaller baseline changes the hardware hurdle.
- Benchmark the proposed replacement.
Run the same acceptance tests against the exact model, quantization, hardware, and serving stack you would deploy. Include long prompts and busy periods, not just the easiest demo.
- Price the whole operating model.
Get a real bill of materials or rental quote. Add power, support, storage, deployment time, ongoing labor, and the residual API bill. Decide what happens when the endpoint fails.
- Stress-test the conclusion.
Reduce the API baseline. Increase labor. Lower the share of spend replaced. Shorten the useful hardware life. If the purchase works only in the optimistic case, preserve the option to wait.
When local can win.
Local inference can make sense when an affordable model already passes a valuable, repeatable workload; when volume is predictable; when suitable hardware is already available; or when offline operation and control over data processing are requirements. Those requirements can justify a cost premium. Name that premium instead of disguising it as savings.
Infrastructure ROI is only one layer of AI ROI.
This calculator asks whether owning infrastructure beats an API baseline. It does not prove that the AI project itself is valuable. Business ROI needs a second comparison: the process with AI versus the process without it.
Price time recovered only when it can be put to useful work. Add review and integration costs. Attribute revenue conservatively. A system can cut inference spending while increasing total delivery cost; a more expensive model can pay for itself by resolving a harder problem.
My conclusion is still conditional. For a small team replacing demanding, general-purpose AI workflows, hardware ownership faces a high bar. For a focused workload with the right model and enough demand, the result can flip. I want the machine on my desk to win. I want the evidence to decide.
06 / OPEN MATH
Methodology, assumptions & sources.
The calculator uses constant monthly demand and prices. It compares undiscounted cash outlay, not accounting depreciation or net present value. A month is 730 hours, making a year 8,760 hours. All figures are US dollars.
- Monthly API cost
- Actual monthly bill, or input million tokens × blended input rate + output million tokens × output rate. Blended input rate weights ordinary and cached-input prices by the cache percentage.
- Monthly owned cost
- Wall power in kW × powered hours × electricity price + other operations + labor hours × labor rate + remaining API bill.
- Monthly rental cost
- GPU count × price per GPU-hour × billed hours + rental operations + remaining API bill.
- Simple payback
- (Hardware + one-time setup) ÷ (API monthly cost − owned monthly cost). No payback when monthly savings are zero or negative. No upfront cost and positive savings means immediate savings.
- Owned-hardware ROI
- ((API monthly cost − owned monthly cost) × months − upfront cost) ÷ upfront cost × 100%. Undefined when upfront cost is zero.
- Break-even baseline API bill
- (Upfront cost ÷ months + owned operations) ÷ share of API spend replaced. Undefined at zero replacement.
The model excludes tax, financing, resale value, replacement hardware, future pricing changes, and downtime losses. It assumes any displaced workload can be handled by the proposed setup. It does not estimate GPU capacity or infer quality from hardware price. The two self-hosted options use the same replacement percentage for comparability; change it when evaluating a different plan.
Default operations of $20,000/year already cover the original example’s operating estimate, so power and labor start at zero to avoid double counting. The workstation and existing-hardware presets are hypothetical starting points. Their rental rates and API token rates are illustrative. Replace them with your own invoice, quote, and measurements.
- Jedidiah Weller: Running frontier AI locally still doesn’t add up — the original argument and team-server assumptions.
- Lambda GPU instance pricing — eight-GPU H100 SXM example; rate checked November 2, 2026.
- vLLM: Optimization and tuning — serving-memory and throughput considerations.
Published and sources checked November 2, 2026. This page does not fetch live prices. Verify current quotes before making a purchase.
COMMON QUESTIONS
AI cost comparison, without the shortcuts.
Is running an LLM locally cheaper than using an API?
It can be, especially for a narrow workload on hardware you already own. Compare the API spend you actually eliminate with hardware, setup, power, support, labor, and any remaining API bill. If monthly operating costs exceed the spending eliminated, there is no simple financial payback. Equal model quality and sufficient serving capacity must be tested separately.
How do I calculate LLM API costs?
Multiply monthly input tokens in millions by the input price per million, then add output tokens in millions multiplied by the output price per million. Price cached input separately when applicable. Include retries, billed reasoning tokens, and any tool, cache-write, or storage charges. The calculator’s token mode covers input, output, and cached-input charges; bill mode can capture the full invoice.
What is the difference between GPU utilization and billed hours?
Utilization describes how much work the hardware is doing. Billed hours measure how long you have rented it. An allocated GPU can be idle and still cost money. Lowering rental hours saves money only if the service can tolerate the reduced availability, startup time, and changed capacity.
Does this calculator compare equivalent model performance?
No. It compares the cash costs of your assumptions. Benchmark model quality, context length, latency, concurrency, and accepted-task throughput before treating any option as a replacement. A configuration that loads a model is not automatically a usable production endpoint.
How should I measure AI ROI?
Separate infrastructure savings from business value. This calculator’s owned-hardware ROI is net savings versus the API baseline divided by the upfront hardware and setup cost. Business ROI also needs evidence of time recovered, revenue created, or losses avoided, less integration, review, and operational costs. Time saved is not automatically cash saved.