如何在 robots.txt 封鎖 AI 訓練爬蟲,同時讓 ChatGPT Search 繼續引用你的網站
How to Block AI Training Bots in robots.txt (and Keep ChatGPT Search Citing You)
網站管理者可按爬蟲用途設定 robots.txt,封鎖 AI 訓練爬蟲,同時保留供 AI 搜尋引用的索引爬蟲。指南將 AI 爬蟲分為訓練、搜尋索引及用戶觸發抓取三類,並指出封鎖 Google-Extended 不會令網站從 Google Search、AI Overviews 或 AI Mode 中移除。
AI companies read your site for three different reasons, and robots.txt lets you treat each one differently. Most "block AI bots" advice lumps them together, which can quietly remove you from AI answers you'd want to be in.
Three kinds of AI bots
-
Training crawlers collect pages to train models:
GPTBot(OpenAI),ClaudeBot(Anthropic),Google-Extended(Gemini),Applebot-Extended,meta-externalagent,Amazonbot,CCBot(Common Crawl) andBytespider(ByteDance). -
AI search crawlers index pages so assistants can cite you:
OAI-SearchBot,Claude-SearchBotandPerplexityBot. -
User-triggered fetchers open one page because someone asked about it:
ChatGPT-User,Claude-UserandPerplexity-User.
Google-Extended and Applebot-Extended aren't crawlers at all. They're labels Google and Apple check before using pages their normal bots already fetched for AI training.
The policy most small sites want
Block training, keep AI search:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /
User-agent: *
Disallow:
One trap: a bot that finds a group with its own name ignores your User-agent: * rules. If you hide /admin/ in the * group and then add a group for a named bot, repeat /admin/ there.
Google works differently
Blocking Google-Extended does not remove you from Google Search, AI Overviews or AI Mode. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To limit what AI Overviews show from a page, Google points to nosnippet, max-snippet and noindex. Blocking Googlebot removes you from Google entirely, so never do it by accident.
What robots.txt can't do
It's a request, not a lock. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules", because a person asked for the page. Anthropic says all its bots respect it. For scrapers that ignore the file, you need your host's or CDN's bot blocking. And llms.txt blocks nothing.
What big sites do
On November 18, 2026, we checked the robots.txt files of 16 well-known sites. The New York Times, the BBC, Reddit and Amazon block every AI bot we tested. Meta's and ByteDance's crawlers were fully blocked on 12 sites and ClaudeBot on 11, but Google-Extended on only 6. Three Indian news sites, Times of India, Hindustan Times and Mint, fully blocked only Meta's crawler.
If you run a blog, a tool site or a small business, blocking training while keeping AI search is usually the best trade: you stay the source assistants link to, without donating your writing to the next model.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google: Google's common crawlers (Google-Extended)
- Google: AI features and your website
- Google: How Google interprets the robots.txt specification
- Google: Optimizing for generative AI features in Search
- Google: exploring controls for Search generative AI features
- Perplexity: Perplexity crawlers
- Apple: About Applebot
- Meta: Meta web crawlers
- Amazon: Amazonbot and Amazon crawlers
- Common Crawl: CCBot
- IETF: RFC 9309, Robots Exclusion Protocol
- Cloudflare: Block AI bots
- Cloudflare: Declaring your AIndependence (Bytespider)
- Bing: Clearer preferences for training and grounding (Oct 30, 2026)
- llms.txt proposal
This is the short version. The full guide, with every step and source, is on GeekSpark.
來源:DEV Community · dev.to