Vyankatesh 靜態掃描 618 個公開 Agent 技能,63.6% 至少有一項掃描發現
I statically scanned 618 public agent skills. Here's what the scanner actually found
Vyankatesh 使用 SkillSpector v2.12.0 靜態掃描 618 個公開 Agent 技能,63.6% 至少有一項發現,40.1% 至少有一項 HIGH 或 CRITICAL 發現。
Disclosure: I ran this study with an AI assistant (Claude), which wrote the scripts, did the analysis and drafted the paper. The assistant also did the manual spot-check described below, so that part is not an independent human review. Everything here is aggregate: no skill, author or repo is named.
Agent skills are folders of instructions (a SKILL.md) plus optional scripts that coding agents load on demand. The text is read by the model as trusted guidance, and the scripts can run with your permissions. That makes public skill collections a supply-chain surface. So I asked a simple question: how risky do public skills look to a static scanner?
What I did
- Found public GitHub repos by topic (
agent-skills,claude-skills,claude-code-skills), then drew a seeded, stratified sample across four star ranges (0-9, 10-99, 100-999, 1000+). - Collected 622 skills from 387 repos. Nothing was installed, imported or run. Each skill went into its own scratch directory and was only read by the scanner.
- Scanned each with an open-source static scanner (SkillSpector v2.12.0,
--no-llm). 618 scanned fine; 4 timed out and were excluded.
Headline numbers
- 63.6% of skills had at least one finding (95% CI 59.7-67.3).
- 40.1% had at least one HIGH or CRITICAL finding (95% CI 36.3-44.0).
- Median risk score was 11; 36% scored 0; the 90th percentile was 100.
The most common categories were "MCP least privilege" (27% of skills), "excessive agency" (25%), privilege escalation (18%) and data exfiltration (16%). Prompt-injection findings appeared in 6.5% of skills.
The interesting part: score tracks code, not popularity
- Stars and risk were unrelated (Spearman rho = 0.01, 95% CI -0.07 to 0.09).
- Age had a weak negative link (rho = -0.10), which I'd treat as exploratory.
- What mattered was executable surface: skills that ship scripts had a median score of 52; markdown-only skills had a median of 0.
A high score is not a verdict
Of the 25 skills with a CRITICAL finding, 23 were CRITICAL only because of "data flow" findings, usually an environment variable (like an API key) flowing into an HTTP request. In the ones I read, the destination was a fixed first-party API endpoint. The other two were outdated dependencies pinned in lockfiles.
I also read 10 randomly chosen skills. Of the 3 flagged HIGH or CRITICAL, none warranted that severity in my judgment. One "memory poisoning" hit was a mocking library's docs saying "clear call history". That's 3 skills judged by one non-human reviewer, so treat it as a hint that severity labels over-state risk, not as an error rate.
I did not find a skill I'd call intentionally malicious. That is not a claim that none exist: static scanning can't see runtime behaviour, and I didn't estimate how many real problems it misses.
Limitations
- Static only; no false-negative estimate.
- Sampled via GitHub topics, so it's not representative of all skills; strata are equal-weighted, not population-weighted.
- One scanner, one version, LLM component off.
- Skills within a repo are correlated, so intervals are probably a bit too narrow.
- The spot-check is n=10, single reviewer, unblinded, and that reviewer is an AI.
- I didn't analyse topical categories (e.g. "data science" vs "devops").
SkillSpector is a static scanner: it lowers risk but doesn't remove it.
Takeaway
Use static scores to triage capabilities (network, credentials, subprocess, unpinned dependencies), not to decide who's malicious. Be most careful with skills that ship scripts, and read what a skill does with your credentials before you enable it. Larger, behaviourally validated studies are the right way to find out how common genuinely malicious skills are.
Related work I found useful (all on arXiv): Agent Skills in the Wild, "Do Not Mention This to the User", Exploring the Emerging Threats of the Agent Skill Ecosystem, SkillJect.
Preprint-style write-up, not peer reviewed.
By Vyankatesh Rohokale
來源:DEV Community · dev.to


