作者用 skill-be 評測 9 個 Claude Skills，發現其中一個令程式碼變差

A SKILL.md file is a program written in English. The model runs it, and like any program it can have bugs. Most skills have no tests, so the usual way to check one is to read it back and decide that it sounds right.

That was my method too. I maintain mini-fp, a small TypeScript library with Option, Either, Try and Validate, and I wrote nine Agent Skills to teach Claude how to use it. Models have barely seen the library, so the skills have a lot to teach. What I could not tell from reading them was whether the model writes different code with them, and a handful of tries by hand is too few to separate the skill's effect from the model's randomness.

So I built skill-be to measure it, and ran it on all nine skills. One of them made the code worse, in a way I would not have found by reading it.

How skill-be measures a skill

The experiment is the simplest one there is: change one thing and compare.

Each skill gets scenarios. A scenario is a task prompt written like a real request, without naming the API, plus a list of criteria a good answer meets. One of the validate scenarios asks:

Write TypeScript validators for a signup form: name (required, non-empty), email (must contain @), age (must be >= 18). Run ALL validators and return ALL errors at once, not just the first one.

One of its criteria is "Uses Validate (not Either) so errors accumulate instead of short-circuiting".

The model gets every task twice: alone (the baseline) and with the SKILL.md text in front of the task. Each version runs 3 times. A second model, the grader, reads every answer and says YES or NO for each criterion. LLM graders are noisy, so each answer is graded 3 times and the majority wins; I would rather pay for the extra calls than trust a single verdict. A criterion's score is the share of runs graded YES.

The scores then go through a few rules. A skill is NOT READY when a criterion scores 0% even with the skill, when a scenario marked critical scores lower with the skill than without it, or when answers could not be generated or graded. skill-be then exits with 1, so CI can stop a release on it. It works like a unit test where the code under test is a page of English and another model checks the assertions.

Results of the full run

The run covered 9 skills, 46 scenarios and 139 criteria. Claude Sonnet 5.5 wrote the code and Claude Haiku 5.5 graded it: 276 answers and 828 gradings in about 38 minutes.

The average went from 22% to 96%, and I was happy with that for about a minute. The two small numbers at the bottom of the table say more about the skills than the averages do.

The 25 criteria at 100% without the skill are things the model already did. Some of them come from a mistake in my evals: one prompt said "Use sequenceValidate", so the baseline used it too, and that scenario could not show anything the skill adds.

The criterion that got worse belongs to the validate skill, and it is why that skill came out NOT READY.

The validate skill made one thing worse

The scenario:

Express + Prisma signup route. Validate email and password synchronously, and check that the email is not already taken with an async isEmailTaken(email): Promise; report all problems together. When everything is valid, save the user with prisma.user.create(...) and respond 201 with the user; otherwise respond 422 with the list of errors.

Without the skill, the model never imported mini-fp. It wrote plain TypeScript with nothing to unwrap, so it passed the last criterion by default. With the skill, all three answers used the library and accumulated every error correctly, which is what the first row shows. Then all three branched on isInvalid() and unwrapped the result:

if (result.isInvalid()) {
  return res.status(422).json({ errors: result.unwrapInvalid() });
}

const [validEmail, validPassword] = result.unwrap();

That if and unwrap() is the code mini-fp exists to replace. The skill showed how to build a Validate and never showed how to turn one into a response, so at that point the model went back to the most common pattern it knows.

Every sentence in the skill was true. What it lacked was an example, and a missing example does not stand out when you read the skill back. The model's answers showed it right away.

skill-be has a triage command for failing criteria: it asks the model for fresh answers, grades them, and proposes a cause. For this scenario it proposed a skill gap, because matchAsync appeared in one line of the skill, with no example of saving on success and responding with the errors otherwise.

The fix was 17 lines in SKILL.md: a short recipe that ends with matchAsync, and a line in the gotchas section that says "To turn a Validate into a response or a final value, use match/matchAsync, not isValid() + unwrap()". On the next run the three criteria scored 100%, 100% and 100%, with answers like this one:

const checked = await validateEmail(email)
  .combine(validatePassword(password))
  .combineAsync(validateEmailFree(email));

const outcome = await checked.matchAsync(
  async () => {
    const user = await prisma.user.create({ /* ... */ });
    return { status: 201, body: user };
  },
  errors => ({ status: 422, body: { errors } }),
);

Advice for writing skills

1. Compare with a baseline. A score with the skill means little until you know what the model scores without it.
2. Look at each criterion. The 96% average hid the one place where the skill made the code worse.
3. Show the last step. The model followed my skill as far as it went and then fell back on its own habits. The missing part was the end of the workflow, which is easy to leave out because it feels obvious when you write it.
4. Keep API names out of the prompts, so the eval tests whether the skill makes the model reach for them.
5. Read the answers behind a surprising score. An LLM grader can be wrong, and doubting it is fair. Several runs and gradings with a majority vote reduce the noise, and skill-be run --html writes a page with every answer and every grader reply, so you can check the grader yourself.

Trying it

skill-be is open source under Apache-2.0 and published on npm:

npm install --save-dev @ngwes/skill-be

The README builds a complete example project you can copy, and has a second example for skills that are not about code, such as a writing format. It works with Claude Code, pi, opencode, and any command-line agent that reads a prompt on stdin. The nine skills and their evals are in mini-fp-skills.

If you run it on your own skills, tell me what it finds. My guess is that plenty of skills are missing a step like this one.