AI 爬蟲的 robots.txt 設定指南:允許搜尋機器人,封鎖訓練機器人
robots.txt for AI crawlers: let search bots in, keep training bots out
CiteProof 作者介紹一種 robots.txt 設定方式,將 AI 搜尋/助理機器人與訓練機器人分開處理,允許前者並按需要封鎖後者,以保留網站在 ChatGPT、Perplexity 或 Claude 答案中的引用機會。
You can block the bots that train AI models without disappearing from ChatGPT, Perplexity or Claude answers. The trick is to treat AI bots as two different groups and to name each bot explicitly in robots.txt.
Two kinds of AI bots
- Search and assistant bots fetch your pages to answer a user's question, and they are the ones that bring you citations: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity), Claude-SearchBot and Claude-User (Anthropic).
- Training-only bots collect pages to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl).
Blocking the second group is a legitimate choice, and it doesn't cost you citations. Blocking the first group does.
The rules
# Search and assistant bots: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
# Training-only bots: your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
Why name every bot
Under RFC 9309, a crawler follows the group that names it, and the * group applies only if no group names it. So an explicit Allow for each search bot survives even if someone later adds a catch-all User-agent: * / Disallow: /.
Google-Extended is a separate call: blocking it also affects Gemini's answers.
robots.txt is only a request
A CDN or firewall can stop bots before they ever read robots.txt. Since July 2025, Cloudflare blocks known AI crawlers by default on new domains and asks new sign-ups whether to let them through (MIT Technology Review, 1 July 2025). Check your Cloudflare setting, your host's anti-bot protection, any WAF and your security plugins.
Check who really got through
Search your server logs for the bot names. But the name in a request can be faked, so verify the IP: OpenAI publishes its bots' IP addresses, and Google's bots are verified with a reverse DNS lookup.
Your 15-minute checklist
- Rewrite robots.txt with the two groups above.
- Check CDN and hosting bot settings.
- Find at least one verified visit from a search bot in your logs.
Full guide (WordPress included): https://citeproof.co/blog/ai-crawlers-robots-txt-llms-txt-wordpress?utm_source=devto&utm_medium=parasite&utm_campaign=block-gptbot
I'm the founder of CiteProof, which measures whether AI assistants cite a site. The rules above work with or without any tool.
來源:DEV Community · dev.to