RAID 排行榜上，Grammarly AI 偵測器因設定不同排第 2、第 4、第 6 或第 17 名

I spent an afternoon on the RAID leaderboard trying to settle a simple question: where does Grammarly's AI detector actually rank? I got four answers, all of them correct, all of them from the same submitted model on the same day.

Nothing about the model changed between those four readings. I changed two dropdowns.

What RAID is, and why the dropdowns exist

RAID is a shared benchmark for machine-generated-text detectors, built by researchers at the University of Pennsylvania and others and published at ACL 2024. Vendors run their detector over a hidden test set, submit the predictions and get a public row. The paper opens with a complaint that reads like a warning label for the whole category: many detectors "claim to detect machine-generated text with extremely high accuracy (99% or more). However, very few of these detectors are evaluated on shared benchmark datasets."

The dropdowns are the reason a shared benchmark is worth anything. RAID does not report one number, because there is no one number to report. It reports a grid, and the leaderboard lets you pick the cell:

• Metric. AUROC, or true-positive rate at a fixed false-positive rate of 5% or 1%. AUROC is threshold-free. The TPR metrics pin a threshold and ask how much AI text you catch when you have agreed in advance to wrongly flag 5%, or 1%, of human text.
• Adversarial attack. None, or any of eleven evasions: whitespace, synonym swaps, paraphrase, homoglyphs, zero-width spaces, article deletion, alternative spellings and more.
• Domain, decoding strategy, repetition penalty. Eight domains — abstracts, books, news, poetry, recipes, Reddit, reviews, wiki — and the generation settings behind the AI half of the corpus.

Every one of those is a URL parameter. That is the useful part: ?domain=all&decoding=all&repetition=all&attack=all&metric=TPR%40FPR%3D5%25 is a citable artifact. A rank you can hand someone as a link is a measurement. A rank in a sentence is a rumour.

The ordering is not stable under a filter change

The interesting thing is not that the numbers move. It is that the ordering moves, and not uniformly.

Go from the default 5% threshold to the stricter 1% one and Grammarly goes up, from 6th to 4th. That is counterintuitive until you look at what the two columns measure. At 5% you are grading on recall with a generous budget for false alarms, and the top of the table is a traffic jam: four entries sit at a rounded 1.000 and the next several at 0.999. Tiny differences decide the order. Tighten the budget to 1% and the jam breaks up, because holding recall while refusing to flag humans is a harder thing to do and fewer models do it equally well.

Now switch the attack filter from none to all. Grammarly drops to 17th. More to the point, QuillBot — 15th in the default view, below Grammarly — comes out 14th, above it. Two detectors swap places because of a dropdown. If you had cited either ordering as the ordering, you would have been wrong half the time and never known which half.

This is the part worth carrying into your own work. When your table has a dial on it, "we rank Nth" is a claim about the dial, not about the model. Report the setting or report nothing.

Append-only tables make yesterday's claim decay

There is a second failure mode here, and it is the one that actually shipped.

Grammarly's entry on RAID is dated 9 February 2026. Among entries dated on or before that day, it was top of the default view. On the same day the company published a post headed "Grammarly's AI Detector Agent Ranks #1 in Quality." Fair claim, at the time.

RAID is append-only. Entries arrive; nothing recalculates the marketing. Every entry now above Grammarly in the default view is dated April 2026 or later. The model did not get worse — the margins at the top are still under a tenth of a point, 0.999 against Winston AI's 1.000 — but the sentence on the page got old. I checked grammarly.com/ai-detector again today, eight months on: it still says "ranks #1", still says "#1 on RAID", still says "99% detection accuracy."

A rank is a function of the table, and the table has other people's commits in it. If your README says "state of the art on X", you have written a claim with a decay rate, and the thing that decays it is other people's work, not yours. The cheap fix is to date it in the sentence: first on the default view as of February 2026 is still true today, and will still be true in a year.

While you are at it, check whether your own surfaces agree with each other. Grammarly's detector page says the tool was trained on "tens of thousands" of texts; its support guide for the same feature says "hundreds of thousands". The page says it "provides clear explanations for why a phrase may be flagged"; the guide says "it doesn't explain why the text may be flagged." Both are the vendor describing one product.

What the benchmark cannot tell you

The blind spots matter as much as the dials, and they are not hidden — they are in the dataset description.

RAID's AI half comes from eleven model variants: ChatGPT, GPT-4, GPT-3, GPT-2, Mistral, Cohere, Llama, MPT and their chat versions. No Claude. No Gemini. Nothing from the current GPT generation. Its eight domains include recipes and poetry and do not include student essays, which is the single text type most people are actually running through a detector.

So a vendor page can truthfully cite a RAID result and truthfully claim its detector "identifies text from ChatGPT, Gemini, and Claude", and the benchmark confirms exactly one third of that. That gap is why I ran my own seven texts through the free page rather than stopping at the leaderboard: three paragraphs generated with Claude, three human texts written before chatbots existed, and one deliberate mix. It scored the Claude texts 75%, 100% and 100%, cleared all three human texts at 0%, and put the mix at 42% against a true machine-written share of 40.8%. QuillBot's detector, run over the same seven, agreed everywhere except one casual first-person paragraph that Claude wrote start to finish: QuillBot called it 0% AI, Grammarly called it 100%. The full method, the texts and the rest of what the free page does and refuses to do are in my Grammarly AI detector test.

Seven texts is a spot check, not a benchmark, and I would distrust an error rate derived from a dozen pastes — including mine. That is the honest version of the same problem the leaderboard has: every number is a number under conditions, and the conditions are the interesting part.

The rule I am taking away

If you publish a benchmark position, publish the query that produces it. A rank needs four things attached before it means anything: the view, the filter set, the threshold and the date. Drop any one of them and you have written something that was true once, under conditions you did not record, and which the next submission can quietly falsify while your page keeps saying it.