LLM 提示詞和回應要不要記錄？四種做法及作者的選擇

Friday AI Question. This week's question is one every team shipping an LLM feature has to answer, usually after the first weird bug report:

Do you log your LLM prompts and responses?

A. No. We don't store prompts or outputs at all.
B. Metadata only. Model, token counts, latency, cost, finish reason, user/session IDs. No text.
C. Full text, redacted, short retention. Prompts and outputs are kept, with obvious personal data masked, deleted after a fixed window, and access is restricted.
D. Everything, forever. Raw prompts and outputs in the same log pipeline as everything else.

Drop your letter in the comments. Here's how I think about each one.

Why this question matters

An LLM feature fails in ways a status code won't show you. The request succeeds, the answer is wrong, and the only evidence is the text that went in and the text that came out. If you never kept that text, you can't reproduce the bug, you can't build an eval set from real traffic, and you can't explain to a user why the bot said what it said.

But prompts are also where users paste the things they shouldn't: names, emails, account numbers, contract text, health details. The OWASP Top 10 for LLM Applications lists Sensitive Information Disclosure as one of its top risks, and your logs are one more place that information can leak from. Privacy laws such as the GDPR also include a storage limitation principle: personal data shouldn't be kept longer than you need it for its purpose.

So the real question isn't "log or don't log". It's "what do you keep, for how long, and who can read it".

A. No logging

Good: nothing to leak, nothing to delete, an easy privacy story.

Bad: you're debugging blind. Every "the bot gave a weird answer" ticket becomes guesswork, and you have no real-world examples to turn into evals. Fine for a prototype; hard to defend for a feature people depend on.

B. Metadata only

Good: you can still watch cost, latency, error rates and stop reasons, and spot that something changed. This is a solid baseline for every LLM app.

Bad: you'll know that quality dropped but not why. Metadata tells you a request was long and expensive; it doesn't tell you the retrieved context was the wrong document.

C. Full text, redacted, short retention (my pick)

Good: you can reproduce real failures, read traces end to end, and sample production traffic into an eval set, while limiting the blast radius.

What "done properly" looks like:

Redact before you store. Mask emails, phone numbers, IDs and other obvious personal data at write time, not at read time.
Set a retention window and enforce it. Pick a number of days that matches how long you actually debug, and delete automatically.
Restrict access. Prompt logs shouldn't be readable by everyone who can read app logs. Keep them in a separate store with their own permissions and an access trail.
Promote, don't hoard. When a logged example is useful, copy it (cleaned up) into your eval set on purpose. Don't keep the whole stream "just in case".
Tell users. Say in your privacy notice that conversations may be reviewed to improve quality, and offer an opt-out where you can.

Bad: it takes real work: a redaction step, a retention job, separate access controls. Redaction is never perfect, so treat the store as sensitive anyway.

D. Everything, forever

Good: maximum debuggability, zero effort.

Bad: every customer secret anyone ever pasted now lives in your log platform, readable by anyone with log access, kept long after it's useful. That's the option that turns a small incident into a big one.

Where I land

Start with B on day one, because cost and latency metrics are cheap and always useful. Move to C as soon as the feature has real users, because you can't improve what you can't read. Avoid D.

LLM observability tools (Langfuse, for example, which I covered in an earlier post) make the "read traces end to end" part easy. The policy part (what to mask, how long to keep it, who can see it) is still your decision.

References

OWASP Top 10 for LLM Applications: https://genai.owasp.org/llm-top-10/
GDPR Article 5, principles relating to processing of personal data (including storage limitation): https://gdpr-info.eu/art-5-gdpr/
Langfuse documentation (LLM tracing): https://langfuse.com/docs
Microsoft Presidio, open-source PII detection and anonymization: https://microsoft.github.io/presidio/

So, A, B, C or D? And if you picked C, how many days do you keep prompt logs?