跳到正文
DEV Community· Driftproofhq·· 2026-10-19AI 評分49

We tested the 20 most-installed agent skills with and without the skill. The most-installed one came out worse.

We tested the 20 most-installed agent skills with and without the skill. The most-installed one came out worse.

AI 導讀

測試者以 Claude Sonnet 5.5 和 Claude Opus 5.5,對 skills.sh 安裝量最高的 20 個 Agent 技能進行載入與未載入的對照測試;find-skills 在兩個模型上都令答案得分下降,由未載入時的 0.92–0.95 降至載入後約 0.75。20 個技能中,11 個在兩個模型上均明確有幫助;40 組結果中有 10 組因答案不足或波動過大而無法判定。

正文

Driftproofhq

skills.sh ranks agent skills by installs. Installs tell you a skill is popular. They don't tell you whether it changes what the agent actually writes.

So we took the top 20 skills from the skills.sh snapshot of 17 Oct and ran each one twice on the same tasks: once with the skill loaded, once without. Two answering models, Claude Sonnet 5.5 and Claude Opus 5.5, with Claude Haiku 5.5 as the judge. 40 results in all.

The scoreboard

Sonnet 5.5 Opus 5.5
Clearly helped 12 12
Clearly hurt 4 2
Too few answers to tell 4 6

11 of the 20 skills clearly helped on both models.

What hurt

  • find-skills (vercel-labs/skills), number one by installs: clearly hurt on both models. Both times it was the same task, a search request, where answers scored about 0.75 with the skill against 0.92 to 0.95 without. find-skills is a router that points the agent at other skills. We tested its written answer, not the skills it hands off to.
  • reddit-automation (flowkit-labs/skills): clearly hurt on both models, on the ranking task, and on the reply task with Sonnet.
  • grill-with-docs and hyperframes: clearly hurt on Sonnet only.

What helped most

  • hyperframes-registry (heygen-com/hyperframes): 0.43 without to 0.98 with on Sonnet, 0.62 to 0.97 on Opus.
  • prototype (mattpocock/skills): 0.36 to 0.93 on Opus.
  • triage (mattpocock/skills): a perfect 1.00 with the skill on both models, against about 0.7 without.

The ones we couldn't call

10 of the 40 results came back "too few answers to tell". grill-me, tdd and Anthropic's own frontend-design landed there on both models.

That is not "no effect". It means answers varied too much from one draw to the next to say either way, even after up to 10 draws per arm. We'd rather print that than a number that flips on a re-run.

How it works

  • For each skill, three text tasks written from the skill's own description and README, fixed before any run.
  • Each task is answered with and without the skill, 3 to 10 times per arm, and each answer is scored 5 times.
  • "Clearly helped" means that on at least one task the two score ranges (mean plus or minus one standard deviation) don't overlap and the gap is at least 0.05, and no task went clearly the other way. A clear drop on any task wins over a gain elsewhere.

Limits

  • Three text-only tasks per skill, written by us, not by the authors. No tool calls, generated media or deployed services were tested.
  • One judge model.
  • Popularity here is reported installs, not quality.

If you think a task is unfair or a result is wrong, every skill page has a link to ask for a re-run, and every receipt, task suite and retained answer is public.

Disclosure: I maintain Driftproof, the open-source tool that produced these results.

來源:DEV Community · dev.to