如何透過 robots.txt 和伺服器日誌檢查 AI 爬蟲能否讀取網站
How to check whether AI crawlers can read your site (robots.txt and server logs)
這篇教學説明如何透過 robots.txt 和伺服器存取日誌，檢查 AI 爬蟲能否讀取網站。日誌中的 200 表示爬蟲讀取頁面，403 或 429 則表示防火牆或速率限制器拒絕請求；Google-Extended 不會出現在日誌中，因為實際抓取由 Googlebot 執行。作者建議在 CDN 或 WAF 放行相關爬蟲、確保頁面採伺服器端渲染，並一星期後重查日誌。
AI answer engines only recommend or cite what they can fetch. Before you invest in GEO, confirm the crawlers can actually read your site. Here is a quick check.
The user agents to know
- GPTBot – OpenAI's crawler, used to collect content that may be used to train models.
- ClaudeBot – Anthropic's crawler.
- PerplexityBot – indexes pages so Perplexity can cite them in answers.
- Google-Extended – a robots.txt token (not a separate bot) that controls whether Google may use your content for Gemini and AI features.
Note that OAI-SearchBot and ChatGPT-User also exist for search and live browsing, so check them if you want ChatGPT to cite you.
Step 1: read your robots.txt
Open https://yourdomain.com/robots.txt. A blanket Disallow: / under User-agent: *, or a rule copied from a security plugin, often blocks AI bots by accident. To explicitly allow them:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
If you prefer to keep some areas private, add Disallow: /private/ inside the relevant block.
Step 2: check the server access logs
robots.txt is a request, not enforcement, and CDNs or firewalls (Cloudflare bot rules, WAFs) can block bots before they ever see it. Your access log shows what really happens:
grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | tail -n 20
Count hits per bot and status code:
grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
Look at the status codes. 200 means the bot read the page. 403 or 429 means your firewall or rate limiter is rejecting it. No hits at all over a couple of weeks may mean it is blocked upstream, or simply that your site is not yet discovered. Note that Google-Extended will not appear in logs, since Googlebot does the fetching.
Step 3: fix and re-test
Whitelist the bots in your CDN or WAF, keep pages server-rendered (many AI crawlers do not run JavaScript), and make sure your sitemap lists the key pages. Then re-check the logs after a week.
We wrote a longer glossary of overseas-ad and AI-search terms here: https://www.pandawm.com/blog/overseas-advertising-glossary.html
來源：DEV Community · dev.to