預先註冊測試結果：掃描器漏掉逾半自行託管應用程式的安全公告
Pre-registered result - your scanner misses half of self-hosted apps' own security advisories
作者的預先註冊測試發現，自行託管應用程式有 53.0% 的合資格安全公告未能在 OSV 找到任何套件記錄，程式庫組為 0%（p = 0.0057）。結果基於 27 個有公告的應用程式倉庫（909 項公告）及 5 個有公告的程式庫倉庫（11 項公告）；作者表示研究由 AI agents 協助完成，尚未經獨立人員審閲。
Three days ago I posted that 31.6% of security advisories for self-hosted AI tools never reach the database your scanner reads. A pre-registered follow-up then showed that AI wasn't the cause: AI projects were no worse than matched non-AI projects (p = 0.30). The pattern that did show up was self-hosted applications vs libraries. That was exploratory, so I wrote down a confirmatory test before collecting any data, froze the code, and ran it on fresh repositories.
It came back this morning. The split holds.
The test
- Frame: 40 self-hosted applications, the top of the awesome-selfhosted catalogue by stars. 40 libraries, the most-depended-on on npm, PyPI and Go. No repository reused from earlier samples. Selection never looked at advisories.
-
Measure: a published repository advisory, at least 14 days old, is scanner-blind when no OSV record names any package. OSV is searched under its own GHSA ID and under every CVE alias. Dependabot,
npm auditandpip-auditcan only match an advisory that names a package. - Test: two-sided permutation test at the repository level, 10,000 permutations, seed fixed in advance. H1 was supported only if p < 0.05 with applications higher.
The result (snapshot 2026-10-18)
| Group | Eligible advisories | Scanner-blind | 95% CI |
|---|---|---|---|
| Self-hosted applications (27 repos with advisories) | 909 | 53.0% | 49.8–56.3% |
| Libraries (5 repos with advisories) | 11 | 0% | 0–25.9% |
Applications minus libraries is +53.0 points, p = 0.0057. That is still p = 0.02 after dropping the heaviest repo in each group. All 80 repositories were readable, and the OSV canary resolved.
Spot-check it yourself. Vaultwarden's GHSA-h6cc-rc6q-23j4 is 404 on api.osv.dev. Its CVE record exists, but it lists only a Git commit range and no package, so no package scanner can match it to your container.
What didn't come out cleanly
- Within each publication period alone, the difference points the same way but isn't significant. The library side has only 5 repos before May 2026 and 1 after.
- Only 5 of the 40 top libraries publish repository advisories at all, so the library rate is imprecise.
- Part of the application number is review lag. 81.5% of 14–29-day-old app advisories are blind, against 36.8% of those 90+ days old. More than a third of old ones are still invisible.
- Exploratory, not pre-registered: apps whose advisories name a registry package (gogs, Payload, copyparty) are mostly visible. Apps shipped only as images are almost entirely blind: Immich, Stirling-PDF, paperless-ngx, Rocket.Chat and LibreChat are at 100%.
What to do
If you self-host something, don't expect your dependency scanner to warn you about that application's own advisories. Watch its Security tab, or run blindspot.py --package repo:OWNER/REPO. If you maintain an app, naming a package in your advisories is what lets them reach scanners.
Full write-up, data (CC BY 4.0) and code (Apache-2.0), including the pre-registration and both amendments: https://github.com/munimtechnologies/advisory-import-census/blob/main/study/app-vs-library/RESULTS.md
The research, code and this post were made with AI agents working for me. Every number is checked against the run's own output files, but no independent human has reviewed it yet. Corrections are welcome.
— MV2 of Munim, Inc.
來源：DEV Community · dev.to