PII 遮罩工具不等於秘密憑證過濾器：LeakProbe 測試 redact-pii 有 52.6% payload 洩漏

If you ship an LLM feature, you probably put a redaction layer in front of the model to strip sensitive data. Good. There's already solid, independent work measuring how well these tools catch PII — academic benchmarks like LeakageBench and REDACT, vendor comparisons on datasets like ai4privacy. If PII is your worry, that ground is covered.

Here's what isn't measured: secrets. The API keys, tokens, connection strings and private keys that end up in prompts not because a user typed them, but because someone pasted a stack trace, an env dump, or a "why isn't this config working?" blob. Those are arguably worse to leak than a name — and nobody's independently testing whether your redactor stops them.

So I built a small, zero-dependency, vendor-neutral tool — LeakProbe — that fires evasive secret (and PII) payloads at any redactor and reports what survives. Then I ran it against a real, popular OSS library: redact-pii.

Give the library its due

redact-pii is genuinely good at the job it advertises. Against my corpus it redacted 100% of the classic PII — emails, credit cards, IBANs, US SSNs — including spaced and lightly obfuscated variants that a naive regex misses. If you picked it to scrub PII, it earns its place.

Now point it at secrets

Here it gets interesting — the coverage is a patchwork:

Secret	Result
Google API key, Slack token, GitHub fine-grained PAT	Caught
DB / Mongo connection strings (password in the URL)	Caught
Private keys — RSA, EC, OpenSSH, even well-formed PEM	Leaked (100%)
Stripe live key, SendGrid key, AWS secret access key	Leaked
AWS access key id, OpenAI key, classic GitHub PAT	Leaked
JWT / Bearer token inside a JSON blob	Leaked
Moroccan phone / national ID (CIN)	Leaked

This is sneakier than "PII tools are blind to secrets." redact-pii catches some credentials — Google, Slack, GitHub fine-grained tokens, even passwords inside connection strings. That partial coverage is exactly what lulls you into trusting it. Then your private keys, your Stripe live key, and your AWS secret access key sail straight through, and you cannot eyeball which half you got. (Overall on my corpus: 20 of 38 payloads leaked, 52.6%.)

It's not that the library is broken — it's a PII tool, and a private key isn't PII. The failure is the deployment: assuming one redactor covers everything sensitive going into your prompts.

The point isn't this one library

Two takeaways:

"We have redaction" is not a measurement. Coverage is specific. A tool that nails PII can pass 100% of your secrets, and you won't know until one shows up in a log, a trace, or a model provider's retained prompt.
If your users aren't US/EU, a lot of "PII redaction" quietly doesn't apply to them. The Moroccan number and CIN were invisible here.

And these aren't hypothetical — the gaps mirror real, documented issues: streamed-output splits (presidio #2312), silently-dropped locale entities (presidio #2256), and secrets hiding in tool-call arguments (litellm #41265). LeakProbe ships a test for each.

Honest caveats

Testing a PII tool on secrets is unfair to the tool — but it's exactly fair to how these libraries get used in front of LLMs.
This corpus is v1 and not exhaustive.
Leak detection is verbatim-survival; partial masking isn't scored yet.
Independent PII benchmarks exist and are good — LeakProbe isn't claiming to be the first benchmark. Its focus is the uncovered slice: secret leakage, tested against your own redactor, in CI.

Run it on your own stack

LeakProbe is open source, zero dependencies. Write a five-line adapter around whatever you use today and get a per-category leak report.

git clone https://github.com/roshcompanylabs/leakprobe && cd leakprobe
node src/cli.js run --target ./your-adapter.js --threshold 0   # fail CI on any secret that leaks

If you run it and something surprising leaks, I'd genuinely like to hear what.