Why single-tenant GPUs matter for privacy

What multi-tenant inference actually shares

When an inference provider runs multiple customers on the same GPU node — the standard operating model for nearly every hosted API — they are sharing more than a physical rack. They are sharing HBM bandwidth, the CUDA scheduler, the L2 cache, the NVLink fabric and the PCIe bus. Your prompt sits in GPU memory alongside another customer's financial records, patient notes, or proprietary legal documents. The only thing separating them is the GPU's virtual memory mapping — a software boundary that was never designed as a security isolation primitive.

The academic literature on GPU side channels is worth taking seriously. Researchers have demonstrated cross-tenant GPU memory probing, contention-based timing attacks and cache-residency inference across process boundaries on both NVIDIA and AMD hardware going back to at least 2018. The mitigations — primarily Nvidia's Multi-Instance GPU (MIG) partitioning on A100/H100, or vGPU profiles — reduce the surface but don't eliminate it. For the B200 generation, the architectural assurances are stronger, but the fundamental threat model remains: you don't know who else is running code on the same die.

What single-tenant gives you

• Full HBM dedicated to your workload. No partition, no scheduler arbitration, no other tenant's tensors in memory alongside yours.
• Elimination of noisy-neighbor tail latency. Your time-to-first-token and inter-token latency reflect your traffic, not someone else's spike.
• Predictable throughput under load. Your continuous-batching scheduler controls the full memory pipeline without contention.
• Collapse of the side-channel surface. No cross-tenant adversary. No need to trust MIG partitioning as your sole isolation boundary.
• Clean audit posture. Dedicated hardware maps one-to-one to your compliance scope, with no co-tenancy explanation required.

Logical isolation is not physical isolation

Multi-tenant providers often describe their isolation as 'logical' — meaning the GPU driver and CUDA runtime enforce boundaries between processes. That is accurate, but it's a weaker statement than it sounds. Logical isolation is what every hypervisor provides; it is also what every hypervisor escape exploit has targeted. Physical isolation — a GPU node that is yours alone, from power-on to power-off — removes that entire class of vulnerability from the threat model. For regulated workloads, that distinction often determines whether a security review passes or stalls.

This is not a theoretical concern. The HIPAA compliance guide flags hardware isolation as a material control for PHI workloads. PCI DSS scoping similarly treats co-tenancy as a segmentation question your QSA will probe. If you plan to run inference on regulated data, the hardware architecture is not an implementation detail — it's a compliance artifact.

The real-world impact on latency

Privacy isn't the only argument. On shared hardware, your inference latency is a function of the aggregate load across all tenants sharing the GPU's memory controller and compute units. A single neighbor running a heavy batch job can push your 99th-percentile latency into territory your SLA doesn't cover and your users will notice. Dedicated GPUs give you a flat latency curve regardless of what's happening anywhere else, because nothing else is happening on your node.

Single-tenant plus zero retention: the compliance piece

Dedicated hardware and zero data retention are complementary controls. Single-tenancy removes the risk of data leaking to another customer on the same silicon. Zero retention removes the risk of your own data persisting beyond the request that produced it. Together they answer the two most common questions auditors ask about inference infrastructure: who else could access this, and where does the data go after? When both answers are 'no one' and 'nowhere,' the conversation moves to more interesting parts of the stack.

When shared is fine, and when it isn't

Shared GPU inference is perfectly reasonable for public-facing chatbots, content generation, code completion and a hundred other workloads where the cost-per-token tradeoff matters and the data doesn't carry regulatory obligations. The calculus flips when the data is protected health information, cardholder data, trade secrets, privileged legal communications or any dataset governed by a DPA with strict processing constraints. At that point, the hardware architecture is the foundation every other control rests on — and sharing it with strangers is a foundation you can't audit. For a detailed comparison of the two models, see dedicated vs shared GPU inference. For specific deployment patterns, our deployment options page maps the architectures.