Blog
Field notes on private inference
Practical guides on running open-weight models privately — deployment, compliance, performance and the engineering behind zero data retention.
- Security
Why single-tenant GPUs matter for privacy
Multi-tenant GPUs share HBM, schedulers and side-channel surface with unknown code. Single-tenant architecture is the difference between logical isolation and genuine hardware dedication.
· 9 min read
- Performance
B200 vs H100 for LLM inference
How B200's 192GB HBM3e, native FP4, and ~8 TB/s bandwidth compare against H100 and H200 for LLM serving — and why supply, not spec sheets, is the real bottleneck in 2026.
· 10 min read
- Migration
Migrating off OpenAI to open weights
Same OpenAI SDK, different endpoint. A practical migration path off proprietary APIs to open-weight models: shadow first, evaluate carefully, route gradually, cut over cleanly.
· 9 min read
- Compliance
What SOC 2 Type II means for inference
SOC 2 Type II audits a provider's controls over months, not a single day. Here's what the five Trust Services Criteria mean for inference, how to read a report, and what to ask for.
· 9 min read
- Guide
Air-gapped LLM inference architecture
How to run frontier open-weight models with zero external connectivity — local registries, physical-media model updates, and the cryptographic controls for classified workloads.
· 10 min read
- Performance
The real cost of LLM inference at scale
Per-token API pricing looks simple until volume climbs. The real cost lives in GPU-hours, utilization, ops, and egress. Here's how to model break-even and when dedicated wins.
· 10 min read
- Performance
How continuous batching speeds inference
Static batching leaves the GPU idle between requests. Continuous batching fills every forward pass by admitting and evicting sequences at the token level — 3-10× throughput, same output.
· 9 min read
- Compliance
GDPR & data residency for LLM inference
GDPR compliance for LLM inference isn't just about a DPA. It's about controller-processor boundaries, cross-border transfer safeguards and region-pinned data residency — all working together.
· 10 min read
- Guide
Dedicated vs shared GPU inference
Shared GPU inference means queuing behind other tenants' traffic. Dedicated hardware gives you flat throughput, predictable tail latency, and true data isolation. Here's the tradeoff.
· 9 min read
- Performance
Quantization without wrecking quality
LLM quantization unlocks capability, not just savings, when you pick the right method. FP8 is near-lossless. AWQ protects salient weights. FP4 on Blackwell changes what's possible.
· 9 min read
- Security
Does your LLM provider log your prompts?
Most providers log your prompts somewhere — even ones that say they don't train on your data. Here's where the text goes, and the questions to ask in a vendor security review.
· 8 min read
- Compliance
ISO 42001 for AI inference, explained
ISO 42001 isn't just another certificate. It's a lifecycle AI governance framework that maps to the EU AI Act and is fast becoming the procurement standard for enterprise inference.
· 8 min read
- Performance
Speculative decoding for faster tokens
A draft model proposes tokens, the target verifies them — lossless 2-3× latency reduction. How speculative decoding, EAGLE draft heads, and multi-token prediction accelerate inference.
· 8 min read
- Guide
VPC vs on-prem vs air-gapped inference
VPC, on-prem, private cloud or air-gapped — break down the four private LLM deployment models across data boundaries, operational burden, and compliance readiness.
· 10 min read
- Compliance
PCI DSS: running LLMs on payment data
Your LLM provider doesn't touch credit card numbers. Until it does. Here's how PCI DSS scoping works for inference, and how to keep the cardholder data environment contained.
· 9 min read
- Performance
KV cache & paged attention deep dive
KV cache stores attention state to avoid recomputation. PagedAttention allocates it in 16-token blocks like virtual memory — reducing waste from 90% to 4% and raising concurrency 2-4×.
· 10 min read
- Guide
PrivateLink & VPC peering for inference
Route LLM inference traffic over PrivateLink and VPC peering so prompts never touch the public internet. Interface endpoints, private DNS, and airtight data residency.
· 9 min read
- Models
Choosing an open model for regulated AI
Regulated teams have more open-weight model choices than ever. Here's how to compare them on what matters: capability, license, context length, provenance, and where they run.
· 10 min read
- Security
Zero retention vs zero logging
Your provider says they don't train on your data. They might still be logging every prompt to disk. Here's how to disentangle the claims and find the architecture that actually holds.
· 7 min read
- Guide
Deploy DeepSeek-V4 in your own VPC
A step-by-step architecture for running frontier open-weight models on dedicated GPUs inside your cloud account — so prompts never leave your security boundary and nothing is ever logged.
· 12 min read
- Security
What zero data retention really means
"We don't store your data" is easy to claim. Here's the architecture that makes it true, and the exact questions to ask any inference vendor.
· 9 min read
- Performance
300+ tokens/sec from open models on B200s
Continuous batching, speculative decoding and custom kernels — the techniques behind tier-1 throughput on dedicated single-tenant hardware.
· 11 min read
- Compliance
A practical guide to HIPAA LLM inference
BAAs, PHI handling, and why zero retention dramatically simplifies your compliance scope when deploying AI in healthcare.
· 10 min read
Get new posts in your inbox
Engineering deep-dives and compliance guides, roughly twice a month. No spam, unsubscribe anytime.