用簡短 Python 腳本在伺服器日誌中識別 AI 爬蟲
Spotting AI Crawlers in Your Server Logs With a Short Python Script
簡短的 Python 腳本可統計伺服器日誌中常見 AI 相關 user agent 的請求次數和狀態碼，協助確認爬蟲能否存取網站，以及是否遭到封鎖。user agent 字串可以偽造，重要決策應核對供應商公佈的 IP 範圍或使用 CDN 的已驗證機械人功能；每週記錄數據亦有助追查設定變更後的影響。
Before you optimize for AI search, check whether AI systems can reach your site at all. Server logs answer that without any paid tool. This post walks through a small script that counts requests from well-known AI-related user agents and flags blocked responses.
Which user agents to look for
Providers publish their crawler names. Common examples include GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended (a control token rather than a crawler), and Bingbot, which feeds several assistants. Names change, so keep the list in a config file and review it quarterly.
The script
Read your access log line by line, extract the user agent, status code, and path, and aggregate counts. A few lines of Python with a regular expression for the combined log format is enough:
- Compile a pattern for the user-agent substrings you care about.
- For each log line, parse status and path with a regex.
- Increment counters keyed by crawler and status class (2xx, 3xx, 4xx, 5xx).
- Print a table sorted by total requests.
Reading the output
- Mostly 200s: crawlers reach your content.
- A wall of 403s: a firewall, CDN rule, or bot-management setting is blocking them. Decide whether that is intentional.
- Many 301s: redirect chains waste crawl budget; link to canonical URLs.
- Many 404s: crawlers are following old links; add redirects for valuable ones.
Verify before you trust
User-agent strings can be spoofed. For important decisions, verify IP ranges published by the provider, or use your CDN's verified-bot features. Do not block or allow based on the string alone.
Make decisions deliberately
You may want search-oriented crawlers to read your public pages while keeping training crawlers out, or the opposite. Both are legitimate. Write the policy down, then express it in robots.txt and in CDN rules so they agree.
Add context with pages
Group requests by path prefix. If crawlers hit your blog but never your pricing or services pages, check internal links and sitemap coverage. Pages that are never fetched cannot be cited.
Keep a record
Save the weekly counts in a sheet. Changes after a CDN update or a robots.txt edit show up immediately, which makes debugging much faster than guessing.
Next steps
Schedule the script as a weekly job, send the table to the marketing channel, and agree on who responds when blocked requests spike. Ten minutes a week of attention is usually enough to catch an accidental firewall rule before it costs a quarter of visibility.
For context on why crawl access matters as AI assistants break one question into many, read this explainer on query fan-out: notes on query fan-out and AI search crawling.
Published by PandaWM, an overseas marketing team based in Hong Kong.
來源：DEV Community · dev.to