The real cost of LLM inference at scale

The per-token bill is not your real cost

Per-token API pricing works beautifully at low volume. You pay a predictable amount per thousand tokens, zero infrastructure to manage, zero ops burden. But the vendor is pricing in their GPU-hours, utilization gaps, operational overhead, and margin — and charging you for all of it. As your volume grows, the convenience markup becomes a line item worth optimizing. Understanding TCO means looking past the per-token number and modeling what the inference actually costs to run on hardware you control.

What is inside the per-token price

Every inference API provider runs a business on top of GPU clusters. Their costs include hardware depreciation or rental, power and cooling, networking and egress fees, SRE and on-call coverage, model serving infrastructure, and margin. The per-token rate you see is the sum divided by expected token volume, padded for variance. At low volume you are mostly paying for convenience. At high volume, you are paying for idle capacity you are not using but the vendor is still running. The economics of B200 versus H100 hardware shift the equation further because newer GPUs deliver far more tokens per dollar of depreciation.

The break-even zone — around $5,000 to $10,000 per month

Industry patterns across cloud GPU pricing and inference API rates converge on a rough break-even zone: when your monthly inference API spend crosses roughly $5,000 to $10,000, running on dedicated or self-hosted hardware starts to look cheaper on a pure dollar basis. Below that, the API's utilization advantage (multi-tenant batching across thousands of customers) usually wins. Above it, you are paying for throughput capacity that the API vendor amortizes across many customers, and the economics of owning the hardware begin to favor you. The exact crossover depends on your model size, concurrency patterns, and whether you need 24/7 availability or can tolerate cold starts. This is an industry-level estimate; your mileage depends on the specifics.

Where dedicated inference wins on cost

Three conditions tilt the math heavily toward dedicated. First, steady and predictable volume: no cold starts, no burst-idle cycles to amortize. Second, high concurrency: you are batching tens or hundreds of simultaneous requests, not sending them one at a time. Third, compliance requirements that already demand single-tenant infrastructure. When all three apply, the cost argument is straightforward: you are already paying for compliance isolation, and adding inference to that dedicated environment is incremental, not greenfield. The dedicated versus shared GPU comparison matters here because shared infrastructure can mask utilization problems that dedicated makes visible — and fixable.

DIY self-hosting — the costs teams miss

The raw GPU-hours are the easy part to model. What teams routinely underestimate: the engineering time to build and maintain a serving stack (kernel tuning, continuous batching config, model updates and rollbacks), the operational burden of 24/7 uptime (pager rotations, incident response, capacity planning), and the utilization penalty when traffic is bursty (idle GPUs still depreciate). A small team self-hosting inference often discovers that saving $8,000 per month on API calls costs $15,000 in engineering time. The math works better when you already have a platform team managing GPU infrastructure, or when you use a managed dedicated service that runs the ops for you.

How to model it for your workload

Start with three numbers: your average tokens per second across a representative week, the peak-to-average ratio, and your latency budget. From there, size the GPU cluster (model weight size divided by quantization factor, divided by per-GPU HBM, plus concurrency headroom), multiply by the per-GPU-hour cost in your cloud or colo, add a 20 to 30 percent utilization buffer, and compare to your API spend at that volume. For most teams the answer is clear on one side or the other, but you need the numbers, not intuition. A deployment scoping call can turn these estimates into a real quote in minutes.

• GPU-hours consumed — model size, quantization level, tokens per second, concurrency target.
• Utilization rate — what fraction of provisioned capacity actually serves requests versus sits idle.
• Ops overhead — SRE time, on-call coverage, incident response, model versioning and rollbacks.
• Networking and egress — data transfer costs between your application, the inference cluster, and end users.
• Compliance overhead — audit burden, BAA or DPA management, retention controls for regulated workloads.
• Opportunity cost — what your engineering team could build instead of maintaining a serving stack.