Does your LLM provider log your prompts?

The four places your prompt text ends up

Prompt logging is not one thing. It's at least four separate data paths, and most providers use some combination of them. When you read a privacy policy that says 'we may retain data to improve service quality,' it's almost always pointing at one of these:

• Persistent storage. Prompts and completions written to a database or object store — sometimes permanently, sometimes with a retention window like 30 or 90 days. This is the most common form and the easiest to audit, because it's contractual.
• Application logs and distributed traces. Debug-level logging that captures request payloads. A support engineer investigating a failed call can often see the exact prompt that triggered it. These logs typically rotate within days to weeks but are rarely mentioned in the privacy policy.
• Abuse monitoring buffers. Prompts held in a review queue for safety classification or abuse detection. These are human-readable, often retained for days, and are explicitly carved out of most 'no training' commitments.
• Training and evaluation pipelines. Prompts used to fine-tune, preference-tune, or benchmark models. This is the one providers phrase as 'we do not train on customer data,' and it's the one most buyers focus on — while missing the other three.

Why "we don't train on your data" isn't enough

The commitment not to use your prompts for training is the headline most providers lead with. It's the easiest promise to make and the one security reviewers latch onto. But a provider can truthfully say they don't train on your data while still writing every prompt to a log file, retaining completions in a database for 90 days, and routing flagged content through a human review queue. None of those are training. All of them are retention.

This is why zero data retention needs a precise definition. It means none of the four paths above are active for your request content — no storage, no logging, no abuse buffer, no training. The distinction between retention and logging matters because vendors often bundle them under softer language that sounds reassuring without actually constraining behavior.

How to audit what your provider actually does

You can't inspect a provider's internal infrastructure, but you can get most of the answer from three documents and one runtime check. Start with the Data Processing Agreement: it should list every subprocessor who touches inference data and describe the categories of data each one processes. If 'prompt content' appears under a logging or monitoring subprocessor, that's your answer. Next, pull the privacy policy and trust center for data retention schedules — look for language like 'retained for up to N days' or 'used for service improvement' alongside any carve-outs for safety and abuse detection.

Then do the runtime check: examine the response headers. A provider that returns an explicit retention header — for example X-Retention: none — on every response is making a verifiable claim you can assert against in CI. If no such header exists, ask your vendor to add one — it's a trivial engineering change that signals whether they're willing to be accountable at the API level, not just in the contract. For teams migrating off OpenAI, this audit surface is one of the main reasons to move to a deployment model where you can see every byte's path.

Questions to ask in a vendor security review

• Is prompt or completion content ever written to persistent storage, even temporarily? If so, for how long and under what access controls?
• Do prompts appear in application logs, request traces or an observability pipeline? Can a support engineer read my request content during a debugging session?
• Is there a human-accessible abuse review queue, and does it contain full prompt text?
• Are prompts used — directly or indirectly — for model training, fine-tuning, evaluation, or benchmarking? Does this include human preference data derived from my outputs?
• Can I verify retention behavior programmatically, for instance through a documented response header?
• What metadata is retained and for how long? Is it segregated from request content?

The architectural alternative

The only way to answer the logging question with a provable no is architecture: request content exists in volatile GPU memory for the lifetime of one request and no code path writes it anywhere else. That's the invariant single-tenant GPU deployment makes practical — the hardware is yours, the observability pipeline carries metadata only, and the DPA backs both with binding terms. If your provider can't explain what happens to a prompt from arrival to memory deallocation at that level of detail, assume it's being logged somewhere until proven otherwise. See our security overview for how these controls map to the SOC 2 and ISO 27001 frameworks that back them.