B200 vs H100 for LLM inference

The hardware, side by side

• H100 SXM. 80 GB HBM3, 3.35 TB/s bandwidth, 1,979 FP8 TFLOPS, 700W.
• H200 SXM. 141 GB HBM3e, 4.8 TB/s bandwidth, 1,979 FP8 TFLOPS, 700W.
• B100 SXM. 192 GB HBM3e, ~8 TB/s bandwidth, native FP4, 700W.
• B200 SXM. 192 GB HBM3e, ~8 TB/s bandwidth, native FP4 with higher power ceiling, 1,000W.

Memory bandwidth is the binding constraint

LLM token generation is memory-bandwidth bound. Each new token reads the entire model weights plus the KV cache from HBM; tensor cores spend most of their time waiting on data. H200's 4.8 TB/s delivers substantially more tokens per second than H100's 3.35 TB/s on identical compute silicon, and B200's ~8 TB/s roughly doubles it again. The 192 GB capacity is the other half: it fits 140B+ parameter models on a single GPU at FP4, eliminating tensor-parallel overhead for many workloads, and leaves room for larger KV caches — higher concurrency and longer context on the same node. See how these numbers stack up in our 300+ tokens per second benchmarks.

Native FP4 changes the serving math

The B200 introduces hardware-native FP4 with register-level support and a dedicated quantization engine. It is not retrofitted INT4 — it is a new precision format with better dynamic range. For inference, the practical effect is half the memory footprint of FP8 with negligible quality loss, provided the model was trained or fine-tuned with FP4 in mind. Combined with 192 GB of HBM, more models fit on fewer GPUs at higher concurrency. For how quantization more broadly affects output quality, see our guide to quantization and model quality.

H200 is the generation too many teams skip

H200 is the quiet value play of the current market. Same compute silicon as H100, but 141 GB of faster HBM3e — meaning significantly higher throughput on memory-bound LLM workloads without the cost and power premiums of Blackwell. For teams whose models fit comfortably in 141 GB and who do not need FP4, H200 is often the better TCO bet right now. The catch: availability is no easier than B200.

The real bottleneck is supply, not specs

Blackwell is ramping. Demand substantially exceeds supply and will for the foreseeable future. B200 allocation is competitive industry-wide. Even H100, two years after launch, remains tight for new allocations. Rolling out a production deployment means working with a provider who has secured capacity, not just one who lists it on a pricing page. The hardware on the spec sheet and the hardware you can actually provision this quarter are often different things.

Dedicated B100 and B200 at HyperInfer

HyperInfer provisions dedicated, single-tenant B100 and B200 nodes — not virtual GPUs, not shared instances. On dedicated hardware, the memory bandwidth, FP4 compute, and KV-cache pool are yours alone. No multi-tenant scheduler competing for HBM, no noisy neighbors. The numbers on the spec sheet are the numbers you get. For the cost angle, see our analysis of inference at scale. For measured throughput, see our benchmarks.