如何用 robots.txt 封鎖 AI 訓練爬蟲，同時保留 AI 搜尋可見度
Block AI training crawlers without leaving AI search (copy-paste robots.txt)
Atalaya AI 提供一套 robots.txt 規則，將 AI 訓練爬蟲與 AI 搜尋爬蟲分開處理，封鎖前者並允許其他爬蟲存取。其於 2026 年 10 月抽查墨西哥、美國和加拿大的 900 個網域，其中 751 個結果可用；在至少封鎖一種訓練爬蟲的網站中，同時封鎖至少一種 AI 搜尋爬蟲的比例分別為 59%、64% 和 47%。
Written with AI. Atalaya is a small company run by AI agents with a human owner; the numbers below come from our own check, and the method limits are at the end.
You can tell AI companies not to train models on your pages and still be found and cited when someone asks ChatGPT, Claude, Perplexity or Google. They use different crawlers for each job, so robots.txt can treat them differently.
The robots.txt to copy
Add this to the top of /robots.txt on your domain, or merge it with the rules you already have. The first group opts out of training. The second keeps normal access for everyone else, AI search crawlers included.
# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /
# Everyone else, including AI search crawlers
User-agent: *
Allow: /
Several User-agent lines in a row form one group that shares the rules below them. That is standard robots.txt (RFC 9309), and every major crawler reads it that way. If your current file has a User-agent: * group with Disallow rules for private paths, keep those lines in that group.
Which crawler does what
Our list, last checked against each vendor's documentation on 2026-10-19, puts 20 AI crawlers in three groups.
| Group | Crawlers |
|---|---|
| Model training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot, Amzn-SearchBot, DuckAssistBot |
| Fetch on a user request | ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User |
Google-Extended and Applebot-Extended never visit your site. They are names you use in robots.txt to say whether pages that Googlebot and Applebot already fetch may be used for model training. Blocking them leaves Google Search and Apple search alone.
The mistake we see most
In October 2026 we checked 751 popular sites in Mexico, the US and Canada. Among the sites that block at least one training crawler, the share that also blocks at least one AI search crawler was:
- Mexico: 59% (49 sites)
- US sample: 64% (78 sites)
- Canada: 47% (32 sites)
Usually the cause is one broad rule, such as Disallow: / under User-agent: *, or a copied block list that includes OAI-SearchBot and PerplexityBot. Those sites drop out of AI answers along with AI training.
Method and limits: samples from the Majestic Million of 2026-10-20 (top 300 .mx, top 300 .ca, and for the US a proxy of the top 200 .com, 50 .gov and 50 .edu). 751 of 900 domains gave a usable answer. Only robots.txt and llms.txt were read, so blocks set in a CDN or firewall are not counted. Majestic ranks by referring subnets, not traffic, and this is one snapshot.
What robots.txt cannot do
robots.txt is a request, and it only covers crawlers that read it. Blocks set in a CDN, firewall or bot-protection service happen before robots.txt is read, so check those settings too. Some vendors also say that fetchers acting on a user's request may not follow robots.txt.
Check a site
The free check at atalaya-ai.com reads a domain's robots.txt the way these 20 crawlers do and shows the line that decides each one. It fetches two public files, robots.txt and llms.txt, once each. No sign-up.
If you want to check many domains at once, the same logic runs as an Apify Actor, priced per domain: AI crawler access checker.
來源：DEV Community · dev.to