robots.txt、Cloudflare 與 Anubis 如何阻擋 AI 爬蟲及各自的實際作用

A reader asked this under our analytics article, and it is the right question to ask in that order: robots.txt, Cloudflare, Anubis. The three get discussed as alternatives. They are not. One publishes a preference, one enforces a decision at the edge, one makes the client pay for the page. Choosing between them means deciding which of those three things you actually want.

We have fifteen days of access log from a site that went live in October, 9,996 lines of it, and a throwaway Anubis stack on a laptop. Everything below with a number attached comes from one of those two. Where we have no numbers of our own, it says so.

First, who even asks

Before picking a tool, look at what the polite protocol is worth on your own log. Of 9,996 requests, 334 were for robots.txt. Here is who sent them, against how much each crawler took:

The two zeroes in the middle of that table are the interesting part, and they are not what they look like at first glance.

GPTBot fetched our robots.txt zero times in 391 requests. OAI-SearchBot fetched it 35 times - and the two share IP ranges. The same applies at Meta: meta-externalagent, the crawler that collects training data, never asked for the file, while facebookexternalhit, the link-preview fetcher, asked eight times. Google does it too: Googlebot fetched it 17 times, GoogleOther not once, and both run on the same infrastructure.

So the honest reading is not "the training crawlers ignore robots.txt". It is that the fetch happens somewhere else in the vendor's fleet, and the crawler you see in your log is downstream of a cache you cannot observe. Which has a practical consequence: your log cannot tell you whether a robots.txt rule was ever seen, let alone obeyed. If you want to know if a rule works, the only test is whether the disallowed paths stop being requested.

Anthropic is the outlier in the other direction. 78 fetches across 484 requests is one robots.txt check for every six content requests, and the first two requests ClaudeBot ever made to us were /robots.txt and then /, one second apart.

robots.txt: a request, with a thin layer of law behind it

What it costs: nothing. What it does: nothing, mechanically. A disallowed path is still served to anyone who asks.

What changed recently is the paperwork behind it. Article 53(1)(c) of the EU AI Act requires providers of general-purpose AI models to put in place a policy to identify and respect reservations of rights expressed under Article 4(3) of the copyright directive, "including through state of the art technologies". The accompanying Code of Practice spells one of those technologies out: signatories commit to honour robots.txt in line with RFC 9309.

Read the scope carefully before you rely on it. That obligation lands on providers of general-purpose models, and the Code is voluntary and signed. A scraper that is not a model provider is outside it, and what counts as a valid reservation of rights is still a question for the courts. The Commission's own FAQ lists several other machine-readable mechanisms beside robots.txt.

The practical value is real but narrow: a robots.txt with explicit AI directives is cheap, and it is the artefact you point at if you ever need to show that you expressed a reservation. It is evidence, not a wall.

Cloudflare: enforcement, if your DNS is already there

This is the section where we have no first-party numbers, and we would rather say that than dress up the documentation. Our own blog does not sit behind Cloudflare, so nothing below was measured by us.

From Cloudflare's docs, AI Crawl Control shows which AI services are crawling you, lets you set allow or block rules per crawler, tracks robots.txt compliance, and exposes a pay-per-crawl option that is in private beta. The managed robots.txt feature generates the directives from your chosen policies and prepends them to whatever your origin already serves. Both are documented as available on all plans.

The part that matters architecturally: this is the only one of the three options that blocks a request before it reaches your server, and the only one where somebody else maintains the list of who is who. That is the whole trade. You get enforcement without running anything, and in exchange the decision about what counts as an AI crawler is made by a third party, in front of your site, for every visitor.

Anubis: make the client pay

Anubis is a reverse proxy that puts a proof-of-work challenge in front of the origin. No identity claim is involved: a client that cannot execute the challenge JavaScript does not get the page, whatever its user agent says.

We ran v1.28.2 in front of an nginx origin on a laptop and probed it. With the bundled default policy:

Three things in that table are worth knowing before you deploy it.

The deny page comes back with HTTP 200, not 403. If you monitor your site by status code, a fully blocked crawler looks like a healthy one.

A plain, honest non-browser client passes through untouched. That is deliberate - it keeps curl, feed readers and your own monitoring working - but it also means Anubis is not a wall against scraping in general. Anything that does not pretend to be a browser is waved past by default.

And the Googlebot row is a lab artefact worth flagging: Anubis verifies search engines by reverse DNS, which cannot succeed for a request arriving from 127.0.0.1. On a real deployment that row would be a pass, not a challenge. We are showing it because it is the kind of result that looks like a finding and is not one.

What the challenge costs the human

This is the number that decides whether you can use Anubis on a site with real visitors. We drove the challenge with a headless Chromium and timed from navigation to the origin page rendering:

At the shipped default the gate is invisible. Two steps up it is a minute of a spinning page, because each level multiplies the work by sixteen. If you turn that dial because the default "feels low", you will not notice what you did - your own browser caches the clearance cookie for a week - while every first-time visitor pays.

Two deployment traps we hit, both costing more time than the measurement itself:

The DIFFICULTY environment variable did not change the challenge. In v1.28.2 the startup log reports it, the served challenge ignores it, and the number that counts comes from the policy file. We only caught it by reading the difficulty out of the challenge page's own embedded JSON.

Without USE_REMOTE_ADDRESS=true, every single request returned HTTP 500, with [misconfiguration] X-Real-IP header is not set in the log. Anubis assumes something in front of it sets that header. If nothing does, it fails closed on all traffic, not just bots.

Can you even tell who is who

All three approaches eventually rest on identification, so we checked how well that works now. Every crawler request in our log, matched against the vendor's own published list or reverse DNS:

Better than the folklore suggests. The major AI crawlers now publish IP lists you can fetch and check, and ours matched to the request.

Two exceptions. ChatGPT-User sent two requests from addresses outside the published list, which is the one user-triggered agent in the table and the one where a mismatch is least alarming. And meta-externalagent has no reverse DNS and no published crawler list we could find, so the only check available is whether the address sits in Meta's announced network prefixes, which is a much weaker claim: it says the packet came from that company's network, not that the crawler is the one it claims to be.

Pacing, when identity is not available

For anything that neither verifies nor identifies, the behaviour still gives it away. The interval between consecutive requests from the same crawler, over fifteen days:

That is a clean split, and it is not about politeness. Google fetched more pages than anyone (541 and 520 URLs) and did it with a median gap of over two minutes, because crawl pacing is a solved problem it has had decades to solve. The training crawlers took the site in one pass, as fast as it would serve.

The practical use: on a quiet site, a histogram of inter-arrival gaps separates a crawl from an audience without reading a single user agent. A spike concentrated in one sub-second bucket is a machine. Humans do produce tight bursts - tab restores, prefetch - but they do not sustain them for 390 requests.

So what do you choose

They stack, and that is the sane configuration: robots.txt says what you want, one of the other two enforces whatever part of it you care about enough to enforce, and the log tells you whether any of it worked.

What we run on this site: robots.txt, and nothing else. We publish articles to be read, including by whatever summarises them later. The reason we built the Anubis stack at all was to be able to put a number on the alternative, and the number says the tool is fine and the difficulty knob is the hazard.

What this cannot tell you

One site, one niche, one fifteen-day window, 9,996 requests. A site with real commercial content attracts a different class of scraper, the kind that does not announce itself and does not appear in any of these tables, and against that population every number above is irrelevant.

The Anubis measurements come from a laptop against localhost: no network latency, one client, and a CPU that is faster than the median phone. Treat the difficulty curve as the shape of the cost, not its absolute value.

And the hole we cannot close with logs at all: fetching robots.txt is not obeying it, and not fetching it is not ignoring it. Everything in the first table is a measurement of requests, not of consent.

Originally published at news.rozoom.app.