如何在 nginx、Apache 和 Cloudflare 封鎖 AI 爬蟲
How to Block AI Crawlers in nginx, Apache and Cloudflare
WebDecoy 的指南介紹如何在 nginx、Apache 和 Cloudflare 封鎖 AI 爬蟲,並以 curl 測試封鎖回應是否為 403,以及一般訪客和 Googlebot 是否仍返回 200。
Originally published on webdecoy.com.
robots.txt asks AI crawlers to stay away. A server rule makes them. This guide gives you working configs to block AI crawlers in nginx, Apache and Cloudflare, shows how to test each one, and covers the scrapers that hide their name.
Which AI crawlers to block
Most sites block AI training crawlers and keep AI search crawlers, so ChatGPT search, Perplexity and Claude can still cite and link to them. The configs below use this training list:
| User-Agent token | Operator |
|---|---|
GPTBot |
OpenAI |
ClaudeBot |
Anthropic |
CCBot |
Common Crawl |
Bytespider |
ByteDance |
Meta-ExternalAgent |
Meta |
Amazonbot |
Amazon |
cohere-ai |
Cohere |
AI2Bot |
Allen Institute for AI |
To leave AI search as well, add OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
Two names you will see in robots.txt guides do not belong here: Google-Extended and Applebot-Extended. They are robots.txt tokens that control AI training use, not crawlers, and they never appear in a User-Agent header. Disallow them in robots.txt.
None of these tokens match Googlebot or Bingbot, so search engines are unaffected.
nginx
Add a map in the http block, usually in /etc/nginx/nginx.conf or a file in /etc/nginx/conf.d/:
map $http_user_agent $block_ai_crawler {
default 0;
"~*(gptbot|claudebot|ccbot|bytespider|meta-externalagent|amazonbot|cohere-ai|ai2bot)" 1;
}
~* makes it a case-insensitive regular expression. Then, in each server block you want to protect:
server {
# ...your existing config...
if ($block_ai_crawler) {
return 403;
}
}
Test and reload:
sudo nginx -t && sudo systemctl reload nginx
The map is evaluated only when the variable is used, so it costs almost nothing per request.
Apache (.htaccess)
Add this near the top of the .htaccess file in your site root. It needs mod_rewrite, which most hosts enable:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent|Amazonbot|cohere-ai|AI2Bot) [NC]
RewriteRule .* - [F,L]
</IfModule>
[NC] ignores case and [F] answers 403 Forbidden. On WordPress, put it above the # BEGIN WordPress block so WordPress never sees these requests.
If you manage the server config rather than .htaccess, Apache 2.4 can do the same without rewrites:
SetEnvIfNoCase User-Agent "(GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent|Amazonbot|cohere-ai|AI2Bot)" block_ai_crawler
<Location "/">
<RequireAll>
Require all granted
Require not env block_ai_crawler
</RequireAll>
</Location>
Cloudflare
If your site runs through Cloudflare, block AI crawlers at the edge so they never reach your server.
Option A: a custom WAF rule
- In the Cloudflare dashboard, open your domain, then Security > WAF > Custom rules, and create a rule.
- Choose Edit expression and paste:
(lower(http.user_agent) contains "gptbot")
or (lower(http.user_agent) contains "claudebot")
or (lower(http.user_agent) contains "ccbot")
or (lower(http.user_agent) contains "bytespider")
or (lower(http.user_agent) contains "meta-externalagent")
or (lower(http.user_agent) contains "amazonbot")
or (lower(http.user_agent) contains "cohere-ai")
or (lower(http.user_agent) contains "ai2bot")
- Set the action to Block and deploy.
This works on the free plan. Use contains as shown: the regex operator matches needs a higher plan.
Option B: Cloudflare's AI bot setting
Cloudflare also has its own one-click setting to block AI bots, and AI Crawl Control for managing them. Both use Cloudflare's list of AI crawlers. Use the custom rule when you want the exact same list on Cloudflare and on your own servers. Our comparison with AI Crawl Control covers the differences.
Test your block
Send a request with an AI crawler's User-Agent and check for 403:
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yoursite.com/
Then confirm normal visitors and search engines still get 200:
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yoursite.com/
Over the next days, check your access logs for 403s to AI crawlers:
grep -iE "gptbot|claudebot|ccbot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c
Scrapers that hide their name
Every rule above matches the name a crawler sends. The large AI companies send their real names, so these rules stop them. Scrapers that copy a Chrome User-Agent pass straight through, along with any crawler you haven't listed yet.
That is the gap WebDecoy closes:
- Up-to-date crawler list. WebDecoy recognises over 200 crawlers and AI agents by name, and the list updates without you editing configs.
- Detection beyond the name. Hidden trap links and decoy pages that only bots follow, plus request fingerprints, catch scrapers that pretend to be a browser.
- Fake crawlers caught. Requests claiming to be Googlebot or Bingbot are checked against the operator's published addresses.
- Runs where you do. A one-click Cloudflare edge sensor, a WordPress plugin and Next.js middleware.
Quick checklist
- Disallow AI crawlers in robots.txt, including
Google-ExtendedandApplebot-Extended. - Add the server rule for nginx, Apache or Cloudflare.
- Decide on AI search crawlers: keep them to stay in AI answers, or add them to the list.
- Test with
curl: 403 for GPTBot, 200 for browsers and Googlebot. - Add detection for scrapers that hide their name.
來源:DEV Community · dev.to