Quantization without wrecking quality

Why quantize

Running a 400B+ parameter model at full precision eats VRAM. A single DeepSeek V4 Pro at BF16 needs hundreds of gigabytes just for weights, before you account for the KV cache or a second concurrent request. Quantization shrinks that footprint by 2x to 4x. That means more concurrent sequences on the same hardware, lower cost per token, and smaller clusters to maintain. It is the difference between a proof of concept and a production service — but only if you do not degrade the model along the way.

FP8 — near-lossless and production-ready

On H100 and B200 GPUs with native FP8 tensor cores, 8-bit floating-point quantization is the first method you should reach for. Empirical results across MMLU, HumanEval, and GSM8K show less than 1% degradation versus BF16 for most open-weight models when calibration is done properly. FP8 preserves the dynamic range that integer formats lose, which is why it works well out of the box. If your hardware supports it, there is rarely a reason to skip it. For the models HyperInfer serves, FP8 is the default because it is both safe and fast.

INT8 SmoothQuant — taming activation outliers

Integer quantization (INT8) is more aggressive than FP8: it maps values to a fixed integer grid, which works fine until activation outliers blow out the scale. A single outlier channel can force the entire tensor into a quantization range where most values collapse to zero. SmoothQuant addresses this by mathematically smoothing activation outliers into adjacent weights before quantization, redistributing the difficulty so neither side of the matmul suffers alone. For hardware without native FP8 support, it is the practical bridge to 8-bit inference.

AWQ — four-bit that protects what matters

Dropping from 8 bits to 4 per weight is where quality risk becomes real. Not all weights are equally important, and AWQ (Activation-aware Weight Quantization) exploits this: it identifies the roughly 1% of weights that correspond to large activation channels and protects them at higher precision while quantizing the rest. The result is near-lossless 4-bit inference for most models, roughly 4x memory savings with quality that stays close to FP16 baselines. It is not free — calibration requires a representative dataset — but the tradeoff is the best in class for 4-bit.

FP4 on Blackwell, and when quantization goes wrong

The B200's Blackwell architecture introduces native NVFP4 support with double the throughput of FP8. Four-bit floating-point is less forgiving: a poorly calibrated FP4 quant can drop 5 to 10 points on reasoning benchmarks relative to FP8. With careful per-channel or block-wise calibration on a representative dataset, the best implementations hold most of the FP8 quality while cutting memory requirements in half again. The KV cache side also benefits: FP8 KV-cache quantization roughly doubles concurrent sequences per node without measurable quality loss, and paged attention makes cache management efficient at scale. But not every path is safe. Low-batch INT4 weight quantization can hurt more than benchmarks suggest, especially on reasoning-heavy workloads. The only way to know is to benchmark your quantized models against your actual tasks. Run your eval suite. Compare lm-eval or HELM scores to the BF16 baseline on what your users actually care about. If degradation exceeds a point or two, step up a precision level. The cost is rarely worth the regression.

A practical quantization playbook

• FP8 — near-lossless on H100/B200; the first choice, no reason to skip on supported hardware.
• INT8 SmoothQuant — bridges FP16 to INT8 by smoothing activation outliers; use when FP8 hardware is not available.
• AWQ — 4-bit with salient-weight protection; best-in-class 4-bit quality, roughly 4x memory savings.
• FP4 / NVFP4 — Blackwell-native; cuts memory in half again, but calibration quality decides whether quality holds.
• KV-cache FP8 — doubles concurrency per node; reliable and widely adopted across serving frameworks.
• Always measure. Run your eval suite against the quantized model before shipping. The regression you do not catch is the one that hurts.