Humanizer 約有 60,000 個 GitHub 星標；ai-slop-linter 在 12 項公開任務中高出 Humanizer 2.67 分

I wanted to know whether a linter for AI writing tells helps an agent rewrite a file better than a good prompt does. So I ran a benchmark with 11 arms: the linter packaged as a Claude Code plugin, six other Claude Code skills that claim to remove AI-sounding prose, and Claude with no plugin. Three of those skills (Humanizer, unslop and everwrite) were also run a second time with the prompt calling them by name, which is where the count of 11 comes from. Each arm ran each task three times on Sonnet 5.5.

The short result: on 12 public tasks the linter scored 58.67 of 60 and the best other arm scored 56.00. On 6 held-out tasks, written by an author who had not read the code, four arms scored 29.67 or more of 30, so that set does not separate them.

What was measured

Each task asks an agent to edit a real file in place: a French launch note, a README intro, a pull request description, release notes, a mixed-language guide, a repository set-up. A task scores only what the file contains afterwards, so talking about the problem earns nothing. Each task also carries facts a careless rewrite would break, such as numbers, names, code and links, and scripted checks look for them. Some criteria need judgment, and those went to a blind grader: outputs of all arms were shuffled and anonymised, and the grader gave its reason before its verdict, five votes per cell. Of 99 blind cells, 92 were unanimous.

The other six tools are Humanizer (blader/humanizer, about 60,000 GitHub stars per the GitHub API on 2026-10-24), unslop, everwrite, slopless, avoid-ai-writing and slop-cop. The protocol was written and committed before the first cell ran. A total below is the sum over tasks of the mean over the three runs, with a maximum of 5 points per task. The result applies to those tasks only and ranks nothing in general.

Public tasks

The difference between the linter and the best other arm, Humanizer called by name, is 2.67 points, with a 95% bootstrap interval of [1.00, 4.33]. The lower bound is above zero.

It did not win every task. On h11 and h15 it scored 4.67 where the best rival scored 5.00, and on h18 it tied at 4.67. The benchmark's stricter per-task rule (rule B) fails on those three tasks.

Held-out tasks

Six further tasks were written by an author, a Sonnet agent, who had seen no code of this repository. The person running the measurement had not read them before it.

Here the linter is 0.33 behind unslop, with an interval of [-1.00, 0.00], so the claim that it is first does not hold on this set. It lost that 0.33 on one task, ho1, where it scored 4.67 against 5.00. Four arms sit at 29.67 or higher and even the no-plugin baseline reaches 28.67, so these tasks cannot separate the top arms. Across both sets: first on the public set, and within 0.33 of the best of the 11 arms on the held-out set.

Other models

I also ran Haiku 5.5 and Opus 5.5 once per task, with only the linter, the best other arm of each set and the no-plugin baseline. These totals are descriptive. With one run per task, a difference of one or two points is inside the run-to-run variation. These runs were not blind-graded, so the public scores use the scripted criteria only and their maximum is 57.

What a prompt cannot do

The linter is a command line tool that calls no model and has no dependencies. Its rules are deterministic: the same text gives the same findings. A rewrite can pass every check and still lose a fact, so the tool also has --compare. It lists each number, version, date, link, quotation, code span and name that appears in one text and not the other, and exits 1 when it finds one. It also reports a qualifier that went missing next to a number, and first-person or claim words that the rewrite added.

This is real output on a two-line example I wrote for this post:

$ npx ai-slop-linter --compare before.md after.md
facts: 1 kept, 4 lost, 0 added
lost   date     3 March 2026
lost   number   1200
lost   qualifier up to 40
lost   version  2.4

The plugin's skill runs this after each rewrite and treats a failing comparison as blocking. --compare compares facts and not wording, so it cannot say whether a rewrite is good.

What it costs

This arm cost 2.43 times the no-plugin baseline per cell at list price, on both sets, because it takes more turns and more tokens. In the public set that was 0.125 US dollars per cell against 0.051. The runs were charged to a subscription, and the whole measurement would cost about 91 US dollars at list price.

What went wrong in the measurement

• The session limit stopped the runs twice. 236 public and 160 held-out attempts failed with usage_limit_reached, were kept and never scored, and the same commands ran them again.
• The public and held-out runs went on at the same time. The held-out runs used a later commit, because the harness refuses a working tree whose benchmark folder differs from the commit. The product files are byte-identical between the two commits.
• The blind grader loop died once and was started again, so for a while two copies wrote vote files. All 165 vote files of the re-grade are valid, and none was dropped by hand.
• A re-grade of 33 sampled blind runs agreed with the first grading on 99.4 percent of criteria. Of 60 sampled cells run a second time, 38 of 40 public and 19 of 20 held-out re-runs fell inside the range of the original three runs.

Try it and check it

npx ai-slop-linter README.md
npx ai-slop-linter --compare before.md after.md

To install it as a Claude Code plugin:

/plugin marketplace add Bubblegunn/ai-slop-linter
/plugin install ai-slop-linter@ai-slop-linter

Every cell, the departures from the protocol and a recompute step are in the round 8 evidence folder. REPRODUCE.md in that folder gives the commands for a re-run, which needs Claude Code 2.1.308, Node 24 and about 9 hours. The tasks, criteria and harness are public in bench/head-to-head/, and the earlier rounds and rulings are in docs/HEAD-TO-HEAD.md. The repository is Bubblegunn/ai-slop-linter, and version 0.2.0 is on npm.