Google-Extended 並非爬蟲：為何 Distill 抓取器仍會遵守其規則

Disclosure: we're Hybrid Forge Lab and we built Distill, the URL-to-Markdown API in this post. There's a free plan and a no-signup demo, so you can check everything below yourself.

Google-Extended shows up in robots.txt like any other user agent:

User-agent: Google-Extended
Disallow: /

But no bot ever sends it. Googlebot keeps fetching the page as before. The token only tells Google whether the content may be used for Gemini. Applebot-Extended does the same for Apple.

So a fetcher that announces itself as DistillBot could skip that group and be technically right: it isn't Google-Extended. We don't skip it. Here's why, with numbers.

What 37 robots.txt files say

On 27 October 2026 we downloaded the robots.txt of 37 big sites:

• international news: BBC, The New York Times, The Guardian, the FT, Reuters, AP, The Verge and others;
• Portuguese news: Público, Expresso, Observador, JN, DN, RTP, SAPO;
• platforms: GitHub, Medium, Substack, Wikipedia.

We ran each one through the same parser Distill uses in production and asked about the home page (/):

• 36 of 37 let User-agent: * in.
• 29 of those 36 refuse at least one AI crawler by name.
• 7 name no AI crawler at all: Wikipedia, Forbes, Substack, The Independent, The Telegraph, DN and RTP.

How often each name is refused:

The second bullet is the one that matters. A scraper that reads only the * group would fetch 29 home pages whose owners said no to AI crawlers by name.

How Distill decides

The parser follows RFC 9309: the longest matching rule wins, * and $ wildcards work, and groups for the same agent are merged. Then:

1. If there's a DistillBot group, it decides alone. That's the site owner talking to us directly.
2. Otherwise the path must be allowed by * and by every group for a known AI crawler. There are 15 of them:
   • OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
   • Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai
   • Google: Google-Extended
   • Apple: Applebot-Extended
   • Perplexity: PerplexityBot, Perplexity-User
   • Meta: Meta-ExternalAgent
   • Common Crawl: CCBot
   • Cohere: cohere-ai
   • ByteDance: Bytespider

This is the core of the production code:

own = self.groups.get(settings.BOT_TOKEN.lower())
if own is not None:
    return None if allowed(own, path) else settings.BOT_TOKEN
for token in ("*", *settings.AI_OPT_OUT_TOKENS):
    rules = self.groups.get(token.lower())
    if rules is not None and not allowed(rules, path):
        return token
return None

And this is what a user gets back for the BBC's news page, from the production API:

{
  "url": "https://www.bbc.co.uk/news",
  "status": "robots_txt",
  "detail": "robots.txt disallows this path for GPTBot",
  "markdown": "",
  "took_ms": 146
}

Across the sample, Distill reads 7 of the 37 home pages and refuses 30: the 29 above, plus Reuters, whose * group blocks /. That's the product working as intended.

So why Google-Extended?

The honest finding first: in our sample, Google-Extended never decided anything on its own. All 18 sites that refuse it also refuse other AI crawlers by name. Taking it off our list would change none of these 37 answers today.

We keep it anyway:

• It's the clearest statement of intent there is. A site that writes User-agent: Google-Extended / Disallow: / is saying "index me, but don't feed me to an LLM". Distill fetches pages only to feed them to LLMs. That's exactly the use they refused.
• The name of the bot is an accident. Usage tokens exist because Google and Apple don't run a separate AI crawler. We go by what the page will be used for, not by who knocks.
• Owners can still let us in. A DistillBot group overrides everything else, so a site that blocks AI in general but is happy for Distill users to read its docs can say exactly that:

User-agent: DistillBot
Allow: /docs/
Disallow: /

User-agent: *
Disallow: /

What being strict costs

Some refused pages are ones the user may well have the right to read. When that happens:

• The response says why: status: "robots_txt" and, in detail, the group that refused.
• We don't retry another way. Our Terms forbid fetching through other means a page that refused DistillBot.
• The same goes for noai in a <meta name="robots"> tag or an X-Robots-Tag header, even when the user sends us the HTML directly.

A few more rules from the same RFC:

• robots.txt is checked on every host, including redirect targets, and cached for up to an hour.
• If it answers 5xx or 429, we treat everything as disallowed and check again after 5 minutes.
• A missing robots.txt (404) means no rules.

The fine print on these numbers

• It's the home page only. Article paths can have different rules, so a site can refuse / and allow /articles/, or the other way round.
• It's 37 big publishers and platforms, not the web. We didn't measure smaller sites, and they may look very different.
• robots.txt files change. These are the files as served on 27 October 2026.

Try it

• For site owners: how DistillBot reads robots.txt, and how to opt out or in.
• No signup: the demo on distill.hybridforgelab.com runs the real API in your browser. Try a site you know and see what its robots.txt says.
• Free plan: Basic is $0 for 15 requests a month, with a hard limit, so it never runs up a bill. Paid plans start at $9 for 10,000 requests.
• Docs and plans: RapidAPI listing.

If you run a site and think we read your robots.txt wrong, tell us. That's a bug we want to fix first.