跳到正文
DEV Community· appierun·· 3 小時前AI 評分64

如何用 Python 逐個 URL 檢查 robots.txt 對 GPTBot、OAI-SearchBot、ClaudeBot 和 PerplexityBot 的放行狀態

Which AI Crawlers Does Your robots.txt Block? Check GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot Per URL With Python

AI 導讀

作者提供 ai_robots_check.py，一個只依賴 Python 標準庫的檢查程式，可逐個 URL 顯示多個 AI 爬蟲按 robots.txt 規則獲準或被封鎖的狀態。

正文

Most teams now have an opinion about AI crawlers. Block training, allow search. Or allow everything so the product shows up in ChatGPT and Perplexity answers. Or block it all. The opinion is the easy part. Knowing what your robots.txt actually says to each crawler, for each URL, is harder than it looks, because the rules for picking a group and picking a winning rule are not the ones most people assume.

This post builds a small checker with Python's standard library. You give it a site (or a local robots.txt) and a list of paths, and it prints a table: one row per URL, one column per AI crawler, allowed or blocked. With --why it prints the exact rule that decided each cell. With --sitemap it tests every URL in your sitemap.

Dated note: crawler names and what each one is for were checked on each vendor's own documentation on 3 Nov 2026. Vendors add and rename crawlers, so check their pages before you ship a change.

Disclosure: Appie.run publishes this post. Appie has a free Sitemap to Robots.txt Generator and other sitemap tools, mentioned near the end. The script does not depend on them.

Earlier posts in this series cover validating an XML sitemap and getting AI assistants to describe your product correctly. This one sits between them: before an assistant can describe your pages, its crawler has to be allowed to read them.

Training, search and user fetches are different crawlers

Each big AI vendor now runs more than one crawler, and they do different jobs. Here is what each vendor's own page says, as of 3 Nov 2026:

  • GPTBot: vendor: OpenAI; what the vendor says it is for: Training. Disallowing it indicates content should not be used for training OpenAI's foundation models.
  • OAI-SearchBot: vendor: OpenAI; what the vendor says it is for: ChatGPT search results. OpenAI says changes can take about 24 hours to apply.
  • ChatGPT-User: vendor: OpenAI; what the vendor says it is for: Fetches a user asked for. OpenAI says that because these are user-initiated, "robots.txt rules may not apply".
  • ClaudeBot: vendor: Anthropic; what the vendor says it is for: Training.
  • Claude-SearchBot: vendor: Anthropic; what the vendor says it is for: Improving search result quality.
  • Claude-User: vendor: Anthropic; what the vendor says it is for: Fetches a user asked for. Anthropic says opt-outs are set per subdomain.
  • PerplexityBot: vendor: Perplexity; what the vendor says it is for: Surfacing sites in Perplexity search results. Perplexity says it is not used to train foundation models.
  • Perplexity-User: vendor: Perplexity; what the vendor says it is for: Fetches a user asked for. Perplexity says it "generally ignores robots.txt rules".
  • Google-Extended: vendor: Google; what the vendor says it is for: A control token, not a separate crawler. It covers Gemini training and grounding, and Google says it does not affect Search.

Sources: OpenAI crawlers, Anthropic's crawler article, Perplexity crawlers, Google's common crawlers.

Two things follow. First, "block AI" in one line usually blocks more than you meant, or less. Blocking GPTBot does not remove you from ChatGPT search; that is OAI-SearchBot. Second, the user-fetch agents are documented as not always following robots.txt, so the checker reports what your file says to them, not what they will do.

The four rules that decide the answer

The robots.txt standard is RFC 9309, and Google's robots.txt spec matches it on the points below.

  1. One group per crawler. A crawler uses the group whose User-agent matches its token (case-insensitive). Only if none matches does it fall back to User-agent: *. A specific group replaces the * group. It does not add to it. If you write a GPTBot group with one rule, none of your * rules apply to GPTBot any more.
  2. Groups for the same token merge. Two User-agent: * blocks in one file are combined into one group. This matters more than it used to, because some CDNs now prepend their own block to your file (more on that below).
  3. Longest match wins. Inside a group, the rule with the longest matching path decides. File order does not matter. If an Allow and a Disallow are equally long, Allow wins.
  4. * and $ are wildcards. * matches any run of characters, and a trailing $ anchors the end of the URL. Disallow: /pricing$ blocks /pricing but not /pricing?plan=pro.

Two status code rules matter too. If /robots.txt returns a 4xx, crawlers may treat the site as unrestricted (Google makes an exception for 429). If it returns a 5xx, RFC 9309 says to assume everything is disallowed. And /robots.txt itself is always allowed.

Why not urllib.robotparser?

Python ships urllib.robotparser, and it is fine for simple files. It predates RFC 9309, though, and in Python 3.13 it differs on the rules above:

  • It applies the first rule in file order that matches, not the longest.
  • It does not support * or $ inside paths.
  • It keeps only the first User-agent: * group and ignores later ones.
  • It matches user-agent names as substrings rather than exact tokens.

You can see the first three in its source (Entry.allowance, RuleLine.applies_to and _add_entry). Here is the effect on the sample file used below. This script compares the two:

"""Where urllib.robotparser and RFC 9309 matching disagree on the sample file."""
import urllib.robotparser
from ai_robots_check import AI_CRAWLERS, Robots

text = open("robots-sample.txt", encoding="utf-8").read()
ours = Robots(text)
std = urllib.robotparser.RobotFileParser()
std.parse(text.splitlines())

paths = ["/", "/pricing", "/blog/post-1", "/blog/drafts/x", "/admin/settings",
         "/admin/help/faq", "/docs/guide.pdf", "/shop?session=abc"]
for path in paths:
    url = "https://example.com" + path
    for name, tokens in AI_CRAWLERS.items():
        a = ours.check(tokens, url)[0]
        b = std.can_fetch(name, url)
        if a != b:
            print(f"{path:18} {name:16} RFC 9309: {'allow' if a else 'block':5}  robotparser: {'allow' if b else 'block'}")

Output on Python 3.13.5:

/pricing           OAI-SearchBot    RFC 9309: block  robotparser: allow
/blog/drafts/x     OAI-SearchBot    RFC 9309: block  robotparser: allow
/admin/help/faq    ChatGPT-User     RFC 9309: allow  robotparser: block
/admin/help/faq    Claude-SearchBot RFC 9309: allow  robotparser: block
/admin/help/faq    Claude-User      RFC 9309: allow  robotparser: block
/admin/help/faq    Perplexity-User  RFC 9309: allow  robotparser: block
/admin/help/faq    Googlebot        RFC 9309: allow  robotparser: block
/docs/guide.pdf    PerplexityBot    RFC 9309: block  robotparser: allow
/shop?session=abc  ChatGPT-User     RFC 9309: block  robotparser: allow
/shop?session=abc  Claude-SearchBot RFC 9309: block  robotparser: allow
/shop?session=abc  Claude-User      RFC 9309: block  robotparser: allow
/shop?session=abc  Perplexity-User  RFC 9309: block  robotparser: allow
/shop?session=abc  Googlebot        RFC 9309: block  robotparser: allow

Thirteen disagreements on a 21-line file. /blog/drafts/x is the clearest: the file says Allow: /blog/ then Disallow: /blog/drafts/. The longer rule should win and block drafts, but robotparser stops at the first match and allows them. So the checker below does its own matching.

The script

Save this as ai_robots_check.py. It needs Python 3.8 or later and nothing from pip.

#!/usr/bin/env python3
"""Check which AI crawlers your robots.txt allows, per URL, with the standard library only.

Matching follows RFC 9309 (the robots.txt standard Google also documents):
one group per crawler (most specific user-agent wins, '*' is the fallback),
longest matching rule wins, and Allow wins a tie.
"""
import argparse
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET

# robots.txt tokens, as published by each vendor (checked 3 Nov 2026).
# Each entry lists tokens to try in order; the first one with a group wins.
AI_CRAWLERS = {
    "GPTBot": ["gptbot"],                    # OpenAI, model training
    "OAI-SearchBot": ["oai-searchbot"],      # OpenAI, ChatGPT search results
    "ChatGPT-User": ["chatgpt-user"],        # OpenAI, user-triggered fetches
    "ClaudeBot": ["claudebot"],              # Anthropic, model training
    "Claude-SearchBot": ["claude-searchbot"],  # Anthropic, search quality
    "Claude-User": ["claude-user"],          # Anthropic, user-triggered fetches
    "PerplexityBot": ["perplexitybot"],      # Perplexity, search results
    "Perplexity-User": ["perplexity-user"],  # Perplexity, user-triggered fetches
    "Google-Extended": ["google-extended"],  # Gemini training and grounding
    "Googlebot": ["googlebot"],              # Google Search, for comparison
}
UA = "ai-robots-check/1.0 (+https://example.com/contact)"


def fetch(url, limit=2_000_000):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=20) as resp:
        return resp.read(limit).decode("utf-8", "replace")


def normalize(path):
    """Percent-encode consistently so '/caf%C3%A9' and '/café' compare equal."""
    return urllib.parse.quote(urllib.parse.unquote(path), safe="/?=&*$%:@!,;+~")


def to_regex(pattern):
    """robots.txt path pattern -> regex. '*' is any run of characters, a final '$' anchors."""
    anchored = pattern.endswith("$")
    if anchored:
        pattern = pattern[:-1]
    body = ".*".join(re.escape(part) for part in normalize(pattern).split("*"))
    return re.compile(body + ("$" if anchored else ""))


class Robots:
    def __init__(self, text):
        self.groups = {}      # lowercased token -> list of (allow, pattern, regex)
        self.sitemaps = []
        self._parse(text)

    def _parse(self, text):
        agents, in_rules = [], False
        for raw in text.splitlines():
            line = raw.split("#", 1)[0].strip()
            if ":" not in line:
                continue
            field, value = (s.strip() for s in line.split(":", 1))
            field = field.lower()
            if field == "user-agent":
                if in_rules:          # a user-agent after rules starts a new group
                    agents, in_rules = [], False
                token = value.split("/")[0].strip().lower()  # 'GPTBot/1.4' -> 'gptbot'
                agents.append(token)
                self.groups.setdefault(token, [])  # groups for one token are merged
            elif field in ("allow", "disallow"):
                in_rules = True
                if not value:         # 'Disallow:' with no path means allow everything
                    continue
                for token in agents:
                    self.groups[token].append((field == "allow", value, to_regex(value)))
            elif field == "sitemap":
                self.sitemaps.append(value)
            # Crawl-delay, Content-Signal and any other fields are ignored here.

    def group_for(self, tokens):
        for token in tokens:
            if token in self.groups:
                return token
        return "*" if "*" in self.groups else None

    def check(self, tokens, url):
        """Return (allowed, group, matching rule or None)."""
        parts = urllib.parse.urlsplit(url)
        path = normalize((parts.path or "/") + ("?" + parts.query if parts.query else ""))
        if path == "/robots.txt":
            return True, None, None   # robots.txt itself is always allowed
        group = self.group_for(tokens)
        if group is None:
            return True, None, None
        best = None
        for allow, pattern, rx in self.groups[group]:
            if rx.match(path):
                length = len(pattern.rstrip("$"))
                # longest pattern wins; on a tie, Allow wins
                if best is None or length > best[0] or (length == best[0] and allow):
                    best = (length, allow, pattern)
        if best is None:
            return True, group, None
        return best[1], group, ("Allow: " if best[1] else "Disallow: ") + best[2]


def sitemap_urls(url, limit):
    """Read <loc> URLs from a sitemap or sitemap index (one level of nesting)."""
    root = ET.fromstring(fetch(url, 50_000_000))
    locs = [el.text.strip() for el in root.iter() if el.tag.endswith("loc") and el.text]
    if root.tag.endswith("sitemapindex"):
        out = []
        for child in locs:
            out += sitemap_urls(child, limit - len(out))
            if len(out) >= limit:
                break
        return out[:limit]
    return locs[:limit]


def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("site", help="https://example.com (robots.txt is read from its root)")
    ap.add_argument("paths", nargs="*", help="paths or URLs to test (default: /)")
    ap.add_argument("--robots-file", help="test a local robots.txt instead of the live one")
    ap.add_argument("--sitemap", action="store_true", help="also test URLs from the Sitemap: lines")
    ap.add_argument("--limit", type=int, default=200, help="max sitemap URLs to test")
    ap.add_argument("--why", action="store_true", help="print the rule that decided each result")
    args = ap.parse_args()

    base = args.site.rstrip("/")
    if args.robots_file:
        text = open(args.robots_file, encoding="utf-8").read()
    else:
        try:
            text = fetch(base + "/robots.txt", limit=500 * 1024)  # Google reads 500 KiB
        except urllib.error.HTTPError as e:
            if 400 <= e.code < 500 and e.code != 429:
                text = ""   # RFC 9309: a 4xx means no restrictions
                print(f"robots.txt returned {e.code}: crawlers may treat everything as allowed")
            else:
                sys.exit(f"robots.txt returned {e.code}: treat the whole site as blocked "
                         "until it loads (RFC 9309, 'unreachable')")
        except urllib.error.URLError as e:
            sys.exit(f"could not reach {base}/robots.txt: {e.reason}")
    robots = Robots(text)

    default = [] if args.sitemap else ["/"]
    urls = [p if p.startswith("http") else base + "/" + p.lstrip("/") for p in (args.paths or default)]
    if args.sitemap:
        for sm in robots.sitemaps:
            urls += sitemap_urls(sm, args.limit)
    urls = list(dict.fromkeys(urls))  # drop duplicates, keep order
    if not urls:
        sys.exit("no URLs to test (no paths given and no Sitemap: lines found)")

    short = {name: name.replace("Perplexity", "Pplx").replace("Google-", "G-") for name in AI_CRAWLERS}
    label = {u: urllib.parse.urlsplit(u)._replace(scheme="", netloc="").geturl() or "/" for u in urls}
    width = max(len(v) for v in label.values())
    if not args.why:
        print(" " * width + "  " + " ".join(f"{short[n]:>{len(short[n])}}" for n in AI_CRAWLERS))
    blocked = {name: 0 for name in AI_CRAWLERS}
    for url in urls:
        cells = []
        for name, tokens in AI_CRAWLERS.items():
            ok, group, rule = robots.check(tokens, url)
            blocked[name] += not ok
            if args.why:
                print(f"{label[url]}  {name}: {'allowed' if ok else 'BLOCKED'} "
                      f"(group {group or 'none'}, rule {rule or 'none matched'})")
            cells.append(f"{('ok' if ok else 'NO'):>{len(short[name])}}")
        if not args.why:
            print(f"{label[url]:<{width}}  " + " ".join(cells))
    print(f"\nBlocked counts across {len(urls)} URL(s):")
    for name, n in blocked.items():
        if n:
            print(f"  {name}: {n}")
    if not any(blocked.values()):
        print("  none")


if __name__ == "__main__":
    main()

A few choices worth explaining:

  • AI_CRAWLERS is a plain dict. Each value is a list of tokens to try in order, so you can add a fallback (for example ["googlebot-news", "googlebot"]) or a new crawler without touching the logic.
  • Rule length counts the pattern as written, wildcards included, which matches Google's documented examples (allow: /page against disallow: /*.ph is a tie, so Allow wins).
  • Paths are percent-encoded before comparing, so /café and /caf%C3%A9 are the same rule.
  • Only 500 KiB of robots.txt is read. That is Google's documented limit; anything after it is ignored.
  • Crawl-delay and Content-Signal lines are skipped. They do not change allow or block.

Try it

Here is a sample file with the patterns that usually cause surprises:

# Sample robots.txt with the patterns that trip people up
User-agent: *
Disallow: /admin/
Disallow: /*?session=
Allow: /admin/help/

User-agent: GPTBot
User-agent: ClaudeBot
User-Agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Disallow: /pricing$
Allow: /blog/
Disallow: /blog/drafts/

User-agent: PerplexityBot
Crawl-delay: 5
Disallow: /*.pdf$

Sitemap: https://example.com/sitemap.xml

Run the checker against it:

python3 ai_robots_check.py https://example.com / /pricing "/pricing?plan=pro" \
  /blog/post-1 /blog/drafts/x /admin/settings /admin/help/faq /docs/guide.pdf \
  "/shop?session=abc" --robots-file robots-sample.txt

Output:

                   GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PplxBot Pplx-User G-Extended Googlebot
/                      NO            ok           ok        NO               ok          ok      ok        ok         NO        ok
/pricing               NO            NO           ok        NO               ok          ok      ok        ok         NO        ok
/pricing?plan=pro      NO            ok           ok        NO               ok          ok      ok        ok         NO        ok
/blog/post-1           NO            ok           ok        NO               ok          ok      ok        ok         NO        ok
/blog/drafts/x         NO            NO           ok        NO               ok          ok      ok        ok         NO        ok
/admin/settings        NO            ok           NO        NO               NO          NO      ok        NO         NO        NO
/admin/help/faq        NO            ok           ok        NO               ok          ok      ok        ok         NO        ok
/docs/guide.pdf        NO            ok           ok        NO               ok          ok      NO        ok         NO        ok
/shop?session=abc      NO            ok           NO        NO               NO          NO      ok        NO         NO        NO

Blocked counts across 9 URL(s):
  GPTBot: 9
  OAI-SearchBot: 2
  ChatGPT-User: 2
  ClaudeBot: 9
  Claude-SearchBot: 2
  Claude-User: 2
  PerplexityBot: 1
  Perplexity-User: 2
  Google-Extended: 9
  Googlebot: 2

Look at the /admin/settings row. Most bots are blocked by Disallow: /admin/ in the * group, but OAI-SearchBot and PerplexityBot are allowed. They have their own groups, so the * rules never apply to them. This is the most common surprise in real files: someone adds a group for one AI crawler and quietly drops it out of every general rule. If you want /admin/ blocked for everyone, repeat it in each named group.

Add --why to see the deciding rule:

/pricing  OAI-SearchBot: BLOCKED (group oai-searchbot, rule Disallow: /pricing$)
/pricing  ChatGPT-User: allowed (group *, rule none matched)
/blog/drafts/x  OAI-SearchBot: BLOCKED (group oai-searchbot, rule Disallow: /blog/drafts/)
/admin/help/faq  OAI-SearchBot: allowed (group oai-searchbot, rule none matched)
/admin/help/faq  ChatGPT-User: allowed (group *, rule Allow: /admin/help/)

(Trimmed to the interesting lines.) /admin/help/faq is allowed for *-group bots because Allow: /admin/help/ is longer than Disallow: /admin/. Order in the file plays no part.

When a CDN writes part of your robots.txt

Cloudflare's robots.txt setting docs (last updated 2 Nov 2026) say that when its managed robots.txt is on and your origin already has a robots.txt, Cloudflare puts its managed block in front of your file and serves both as one response. The managed block includes a User-agent: * group with Allow: /, and your own file often has a User-agent: * group too. Under RFC 9309 those two groups merge. Under urllib.robotparser, only the first one counts.

A cut-down version of that shape:

# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no
Allow: /

User-agent: GPTBot
Disallow: /
# END Cloudflare Managed Content

User-agent: *
Disallow: /admin/
Disallow: /checkout/

For ClaudeBot (no group of its own in this cut-down file, so it falls back to *), the checker says /admin/users and /checkout/ are blocked, because the merged * group has Disallow: /admin/ and that is longer than Allow: /. urllib.robotparser says both are allowed, because it never sees your origin's * group. If you audit a Cloudflare-fronted site with the standard library parser, you can get the wrong answer for every URL your own rules cover.

The Content-Signal line is a separate preference signal. Cloudflare's docs say robots.txt compliance is voluntary and the file does not stop access at a technical level. The checker skips the line because it does not change allow or block.

Test every URL in your sitemap

Point the checker at a live site and add --sitemap. It reads the Sitemap: lines from robots.txt, follows a sitemap index one level, and tests up to --limit URLs:

python3 ai_robots_check.py https://yourdomain.com --sitemap --limit 500

This is the version to run after a robots.txt change, a CDN setting change or a site migration. A count like OAI-SearchBot: 37 when you expected zero tells you a group or wildcard is catching pages you wanted in ChatGPT search. Re-run with --why and the paths in question to find the rule.

To keep it from drifting, run it in CI against a staging robots.txt with --robots-file and fail the build if the blocked counts change. A short shell step that diffs the output against a saved copy is enough.

Known limits

  • It reads one host. Rules are per host and per subdomain. blog.example.com has its own robots.txt.
  • It tests what the file says, not what bots do. The user-fetch agents above are documented as not always following robots.txt. Firewalls, bot rules at your CDN and IP blocks are a separate layer the script cannot see.
  • No redirect handling beyond urllib's defaults. RFC 9309 allows up to five redirects for robots.txt. urllib follows redirects, but the script does not count them.
  • It does not cache. Crawlers may cache robots.txt for up to 24 hours, so a fix you just deployed may take a day to be seen.
  • The crawler list is a snapshot. New tokens appear often. Check vendor docs and add them to AI_CRAWLERS.
  • robots.txt is not noindex. Blocking a crawler stops it reading the page. Google's docs say a disallowed URL can still be indexed without a snippet. Use noindex (on a crawlable page) if you need it out of results.

When you do not want to run a script

Appie has free sitemap and robots tools in the browser. Each run asks for a Cloudflare Turnstile check:

  • Sitemap to Robots.txt Generator: you set a user agent, list allow and disallow paths one per line, add sitemap URLs, and download a robots.txt file. The page says to save it at the site root. It writes one group per run, so for a file with separate AI crawler groups, run it once per group and join the results, then check the joined file with the script above.
  • Sitemap URL Extractor: paste sitemap XML or a sitemap URL and get one URL per line. The page says that for a sitemap index it lists the child sitemaps, and you run each child to get its page URLs. That list is a good input for the checker's paths argument.
  • XML Sitemap Validator: paste sitemap XML or give its URL, and the page says it checks the root element, absolute URLs, lastmod, changefreq, priority, duplicates and the 50,000 URL limit. Worth running on the sitemap you list in robots.txt.

If what you want is for AI tools and visitors to get correct answers about your product, the crawler side is only half of it. Appie's AI Chatbots app trains a website chatbot on your URLs, sitemaps, files and Q&A pairs, so visitors on your own site get answers from your pages.

Honest limits of Appie's robots and sitemap tools

  • The generator writes rules; it does not test them. It will not tell you that a named group drops your * rules. That is what the checker is for.
  • One run at a time, with a fresh captcha each time. For hundreds of URLs or a CI check, use the script.

FAQ

If I block GPTBot, will my site disappear from ChatGPT?
Not from ChatGPT search, based on OpenAI's docs. GPTBot is for training. Search uses OAI-SearchBot, which OpenAI says is a separate, independent setting. Users can also still paste or ask for a URL, which uses ChatGPT-User.

Does Google-Extended affect my Google rankings?
Google's crawler page says Google-Extended does not affect inclusion or ranking in Google Search. It is a control token for Gemini training and grounding, with no user agent of its own, so you will not see it in your logs.

Why does my named AI crawler group ignore my Disallow: /admin/?
Because a crawler with its own group does not read the * group at all. Copy the rules you want for everyone into every named group.

What happens if my robots.txt returns a 500 error?
RFC 9309 says crawlers should assume everything is disallowed while the file is unreachable. Google's spec describes its own handling of 5xx errors in more detail. Either way, a broken robots.txt can stop crawling, so monitor it like any other endpoint.

Can I use this for non-AI crawlers?
Yes. Add any token to AI_CRAWLERS, for example "Bingbot": ["bingbot"]. The matching rules are the same for every crawler that follows RFC 9309.

Does Crawl-delay matter?
Some crawlers read it and some do not. Anthropic's article says it supports Crawl-delay, and Google's spec does not list it as a supported field. The script ignores it because it does not change allow or block.

If you want a chatbot on your site that answers from your own pages, you can try Appie's AI Chatbots on the Free plan (500 credits, no credit card) at appie.run.

RG, Founder, Appie.run

來源：DEV Community · dev.to

#教學/實踐#部署/工程#搜尋#OpenAI#Anthropic#Google