跳到正文
DEV Community· Axidort·· 1 小時前AI 評分61

如何檢查網站是否阻擋 AI Agent:robots.txt、WAF 規則與 user-agent

Is your site blocking AI agents? robots.txt, WAF rules, user-agents

AI 導讀

判斷網站能否讓 AI Agent 存取,應依序檢查 robots.txt、伺服器對各 user-agent 的回應,以及 CDN 或 WAF 的機械人規則。內容將機械人分為訓練爬蟲、搜尋爬蟲和用戶觸發的 Agent,建議分開決定是否允許,並避免把 robots.txt 當成強制控制。

正文

Short answer: robots.txt is only half of it. To find out whether AI agents can reach your site, check three layers in order: what your robots.txt says, what your server returns to each AI user agent, and whether your CDN or WAF bot rules challenge them. A permissive robots.txt does nothing if the firewall answers 403 first.

Disclosure: we built Axidort, a tool that scans sites with a real AI agent, and I link to our free guides and scan at the end. The checks below need nothing from us.

1. Know which bots you are talking about

There are three groups, and they deserve separate decisions. This grouping follows Axidort's robots.txt guide, and you should confirm exact names in each vendor's docs because they change (for example OpenAI's bots page).

Group Examples What it does
Training crawlers GPTBot, ClaudeBot, CCBot, Bytespider, plus the control tokens Google-Extended and Applebot-Extended Collect content for model training
Search crawlers OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot Index pages so AI answers can cite you
User-triggered agents ChatGPT-User, Claude-User, Perplexity-User Fetch a page because a person asked

Blocking the second and third groups removes you from AI answers and turns away your customers' assistants. Allowing the first is a business call: it can put your content into future models, and blocking protects content you sell.

2. Read your robots.txt as a bot would

curl -s https://www.example.com/robots.txt

Look for a blanket User-agent: * with Disallow: /, and for per-bot groups. A common intentional setup allows search and user-triggered agents, opts out of training crawlers and keeps carts and accounts out of reach:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /cart
Disallow: /account

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

Two cautions. robots.txt is a request that reputable bots honour, not enforcement. And user-triggered fetches may not consult it the same way crawlers do, so do not rely on it alone either way.

3. Test what each user agent actually receives

This is the check most people skip. Send a request with each user agent string and print only the status code:

for ua in "OAI-SearchBot" "ChatGPT-User" "PerplexityBot" "ClaudeBot"; do
  printf "%s: " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://www.example.com/
done

A 200 is the goal. A 403, 429 or 503 means something in front of your site is refusing that bot, whatever robots.txt says. Axidort's guide calls an Allow line that a CDN or bot-protection layer overrides the most common mistake.

What this test cannot tell you:

  • Anyone can send a GPTBot user agent, so a firewall may treat your curl differently from the real bot, which comes from published IP ranges. Your result is a smoke test, not proof.
  • A 200 for a challenge page is still a block. Compare the response body size, or look for a challenge script, instead of trusting the status code.
curl -s -A "OAI-SearchBot" https://www.example.com/ | head -c 600

4. Check the WAF and bot-management rules

If step 3 shows refusals, look at your edge. Things worth checking in your CDN or WAF dashboard:

  • Is a managed bot or AI-crawler block switched on by default for your plan or zone?
  • Is there a verified-bot list, and are the AI agents you want to allow on it?
  • Are rate limits tight enough to trip a bot that fetches a handful of pages quickly?
  • Does a JavaScript challenge sit in front of product, search or cart URLs, even for declared bots?

To allow a bot safely, verify it instead of trusting its user agent: use the vendor's published IP ranges, your bot-management provider's verified-bot list, or Web Bot Auth, which the Axidort guide describes as an IETF draft. Then keep your protection on for everything else. Letting declared agents through and keeping scrapers out are not opposites.

5. Decide on purpose, then write it down

A small policy beats an accidental one:

  1. Which groups do we allow? (Search and user-triggered, usually yes for a store.)
  2. Which paths are off limits? (Cart, account, internal search results.)
  3. Where is it enforced? (robots.txt for the request, WAF for the rule.)
  4. When did we last test it? Put the curl loop in a monthly check.

FAQ

Does blocking GPTBot hide me from ChatGPT answers?
Not by itself. Per the grouping above, GPTBot is the training crawler; search and user-triggered access are handled by OAI-SearchBot and ChatGPT-User. Check OpenAI's documentation for how they behave today.

Do I need llms.txt?
No. Axidort's own FAQ calls it a cheap extra signal, not a requirement.

My robots.txt allows everything but the test returns 403. Why?
Something earlier than your app, usually a CDN, WAF or bot-protection layer, is deciding first.

Can I trust this test for the real bots?
Use it as a first pass. Check your firewall logs for the real bots' requests, which is the evidence that counts.

Further reading and a free check

The robots.txt for AI agents guide has the full template. If you would rather watch an agent try it, Get my score runs a declared browser agent (AxidortAgentCheck) through search, product, price and stock, and shows where it stopped, in about three minutes. It never orders, pays or logs in. The score is an automated opinion from one scan at one moment, not a verdict on your site.

If you run a business and want a hand with bot rules, you can talk to our team.

來源:DEV Community · dev.to