如何用 50 行 Python 核實日誌中的 GPTBot 等 AI 爬蟲流量

Open any access log today and you'll see a parade of AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot. Before you decide to block them, allow them or celebrate them, it's worth asking a boring question: are they real?

A user agent is just a string. Any scraper can send OAI-SearchBot/1.3 and hope your firewall waves it through. The good news is that the companies behind the major crawlers publish the IP ranges they actually use, so you can check.

Where the official ranges live

Each of these files is plain JSON with a prefixes list of ipv4Prefix / ipv6Prefix entries (all checked on 26 October 2026):

Google also documents a DNS method for Googlebot (reverse lookup on the IP, check the hostname ends in googlebot.com or google.com, then forward-resolve it back to the same IP). The JSON list is simpler for batch checks, so that's what the script uses.

The script

Standard library only. It reads a combined-format log (the nginx and Apache default), picks out requests whose user agent names one of these crawlers, and checks the client IP against that crawler's published ranges.

"""Check which "AI crawler" hits in an access log really come from the crawler's owner.

Usage: python3 verify_ai_bots.py access.log
Reads a combined-format log (nginx/Apache default), finds requests whose user agent
names a known crawler, and checks the client IP against the ranges the owner publishes.
"""
import ipaddress
import json
import re
import sys
import urllib.request
from collections import Counter

RANGES = {
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "GPTBot": "https://openai.com/gptbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "PerplexityBot": "https://www.perplexity.com/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.com/perplexity-user.json",
    "Googlebot": "https://developers.google.com/static/search/apis/ipranges/googlebot.json",
}
LINE = re.compile(r'^(\S+) .*?"[A-Z]+ (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')


def load_networks(url):
    with urllib.request.urlopen(url, timeout=20) as r:
        data = json.load(r)
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
            for p in data["prefixes"]]


def main(path):
    nets = {bot: load_networks(url) for bot, url in RANGES.items()}
    results = Counter()
    with open(path, encoding="utf-8", errors="replace") as f:
        for line in f:
            m = LINE.match(line)
            if not m:
                continue
            ip, url, status, agent = m.groups()
            bot = next((b for b in RANGES if b.lower() in agent.lower()), None)
            if not bot:
                continue
            addr = ipaddress.ip_address(ip)
            real = any(addr in n for n in nets[bot] if n.version == addr.version)
            results[(bot, "verified" if real else "SPOOFED", status)] += 1
    for (bot, verdict, status), n in sorted(results.items()):
        print(f"{bot:16} {verdict:9} HTTP {status}  x{n}")


if __name__ == "__main__":
    main(sys.argv[1])

What the output looks like

I ran it against a five-line sample log I wrote by hand: one request from an IP inside OpenAI's published OAI-SearchBot range, one claiming to be OAI-SearchBot from a documentation-only IP (203.0.113.9), one real PerplexityBot IP, one real Googlebot IPv6 address, and one ordinary browser. The script fetched the live lists and printed:

Googlebot        verified  HTTP 200  x1
OAI-SearchBot    SPOOFED   HTTP 403  x1
OAI-SearchBot    verified  HTTP 200  x1
PerplexityBot    verified  HTTP 200  x1

The browser line is ignored, and the fake OAI-SearchBot is caught. Run it on your real log and the interesting rows are usually:

• verified with HTTP 403 or 429. A real crawler you're blocking, often by accident: a CDN bot rule, a WAF preset, a rate limit. If you want to appear in ChatGPT or Perplexity answers, this is the row to fix first.
• SPOOFED with HTTP 200. Someone borrowing a crawler's name to get past your rules. Safe to block by IP.
• No rows at all for a crawler. It hasn't visited, or your log rotation dropped it. Check robots.txt before assuming anything.

Wiring it into something useful

A few ways I'd use this beyond a one-off check:

1. Run it daily from cron and alert when a verified crawler gets a 4xx on an important page.
2. Feed the ranges into your firewall allowlist instead of trusting user agents. Refresh the lists regularly; they change (the creationTime field in each file tells you when).
3. Split by URL. Add url to the counter key to see which pages each crawler actually fetches. You might find OAI-SearchBot reading your blog but never your pricing page.

What this doesn't tell you

Verified crawler traffic proves an engine can read your pages. It doesn't tell you whether the engine names you when a buyer asks a question. That's a separate check: ask the questions your buyers ask, with search on, and look at who gets recommended and which pages get cited.

That second half is what we build at Presence Scout: it tracks whether ChatGPT, Gemini, Perplexity, AI Overviews and AI Mode name your brand, next to your Google rankings. This short demo shows what that looks like on a real brand:

But start with the logs. If the real crawlers are getting 403s, nothing else matters yet.

I'm Ankit, and I work on marketing at Presence Scout. Feel free to copy and adapt the script.