跳到正文
DEV Community· webdecoy·· 38 分鐘前AI 評分56

如何在 nginx、Apache 和 Cloudflare 封鎖 AI 爬蟲

How to Block AI Crawlers in nginx, Apache and Cloudflare

AI 導讀

WebDecoy 的指南介紹如何在 nginx、Apache 和 Cloudflare 封鎖 AI 爬蟲,並以 curl 測試封鎖回應是否為 403,以及一般訪客和 Googlebot 是否仍返回 200。

正文

Originally published on webdecoy.com.

robots.txt asks AI crawlers to stay away. A server rule makes them. This guide gives you working configs to block AI crawlers in nginx, Apache and Cloudflare, shows how to test each one, and covers the scrapers that hide their name.

Which AI crawlers to block

Most sites block AI training crawlers and keep AI search crawlers, so ChatGPT search, Perplexity and Claude can still cite and link to them. The configs below use this training list:

User-Agent token Operator
GPTBot OpenAI
ClaudeBot Anthropic
CCBot Common Crawl
Bytespider ByteDance
Meta-ExternalAgent Meta
Amazonbot Amazon
cohere-ai Cohere
AI2Bot Allen Institute for AI

To leave AI search as well, add OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.

Two names you will see in robots.txt guides do not belong here: Google-Extended and Applebot-Extended. They are robots.txt tokens that control AI training use, not crawlers, and they never appear in a User-Agent header. Disallow them in robots.txt.

None of these tokens match Googlebot or Bingbot, so search engines are unaffected.

nginx

Add a map in the http block, usually in /etc/nginx/nginx.conf or a file in /etc/nginx/conf.d/:

map $http_user_agent $block_ai_crawler {
    default 0;
    "~*(gptbot|claudebot|ccbot|bytespider|meta-externalagent|amazonbot|cohere-ai|ai2bot)" 1;
}

~* makes it a case-insensitive regular expression. Then, in each server block you want to protect:

server {
    # ...your existing config...

    if ($block_ai_crawler) {
        return 403;
    }
}

Test and reload:

sudo nginx -t && sudo systemctl reload nginx

The map is evaluated only when the variable is used, so it costs almost nothing per request.

Apache (.htaccess)

Add this near the top of the .htaccess file in your site root. It needs mod_rewrite, which most hosts enable:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent|Amazonbot|cohere-ai|AI2Bot) [NC]
RewriteRule .* - [F,L]
</IfModule>

[NC] ignores case and [F] answers 403 Forbidden. On WordPress, put it above the # BEGIN WordPress block so WordPress never sees these requests.

If you manage the server config rather than .htaccess, Apache 2.4 can do the same without rewrites:

SetEnvIfNoCase User-Agent "(GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent|Amazonbot|cohere-ai|AI2Bot)" block_ai_crawler

<Location "/">
    <RequireAll>
        Require all granted
        Require not env block_ai_crawler
    </RequireAll>
</Location>

Cloudflare

If your site runs through Cloudflare, block AI crawlers at the edge so they never reach your server.

Option A: a custom WAF rule

  1. In the Cloudflare dashboard, open your domain, then Security > WAF > Custom rules, and create a rule.
  2. Choose Edit expression and paste:
   (lower(http.user_agent) contains "gptbot")
   or (lower(http.user_agent) contains "claudebot")
   or (lower(http.user_agent) contains "ccbot")
   or (lower(http.user_agent) contains "bytespider")
   or (lower(http.user_agent) contains "meta-externalagent")
   or (lower(http.user_agent) contains "amazonbot")
   or (lower(http.user_agent) contains "cohere-ai")
   or (lower(http.user_agent) contains "ai2bot")
  1. Set the action to Block and deploy.

This works on the free plan. Use contains as shown: the regex operator matches needs a higher plan.

Option B: Cloudflare's AI bot setting

Cloudflare also has its own one-click setting to block AI bots, and AI Crawl Control for managing them. Both use Cloudflare's list of AI crawlers. Use the custom rule when you want the exact same list on Cloudflare and on your own servers. Our comparison with AI Crawl Control covers the differences.

Test your block

Send a request with an AI crawler's User-Agent and check for 403:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yoursite.com/

Then confirm normal visitors and search engines still get 200:

curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yoursite.com/

Over the next days, check your access logs for 403s to AI crawlers:

grep -iE "gptbot|claudebot|ccbot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Scrapers that hide their name

Every rule above matches the name a crawler sends. The large AI companies send their real names, so these rules stop them. Scrapers that copy a Chrome User-Agent pass straight through, along with any crawler you haven't listed yet.

That is the gap WebDecoy closes:

  • Up-to-date crawler list. WebDecoy recognises over 200 crawlers and AI agents by name, and the list updates without you editing configs.
  • Detection beyond the name. Hidden trap links and decoy pages that only bots follow, plus request fingerprints, catch scrapers that pretend to be a browser.
  • Fake crawlers caught. Requests claiming to be Googlebot or Bingbot are checked against the operator's published addresses.
  • Runs where you do. A one-click Cloudflare edge sensor, a WordPress plugin and Next.js middleware.

Quick checklist

  • Disallow AI crawlers in robots.txt, including Google-Extended and Applebot-Extended.
  • Add the server rule for nginx, Apache or Cloudflare.
  • Decide on AI search crawlers: keep them to stay in AI answers, or add them to the list.
  • Test with curl: 403 for GPTBot, 200 for browsers and Googlebot.
  • Add detection for scrapers that hide their name.

來源:DEV Community · dev.to