如何找出拖慢伺服器的 AI 爬蟲，以及應先限制哪一個

A site slows down, someone opens the access log, sees a wall of bot requests, and blocks them. Two days later the site is slow again and the list of bots has changed. The step that got skipped is the one that makes the answer stable: pricing what each crawler actually costs before you change anything.

This is the sequence I'd run on a WordPress box, and it works the same on any LAMP-style stack. It ends with a ranked list and one change to make, instead of a firewall full of guesses.

First check whether the web tier is even the bottleneck

cat /proc/loadavg
top -b -n1 -o %CPU | head -15

If the top CPU consumers are php-fpm and mysqld, HTTP requests caused this and the access log holds the evidence. If you see heavy kswapd, kcompactd, or %wa near 100, memory or disk is the problem — rate-limiting a crawler buys you a few days and nothing else. Look at the PHP-FPM pool as well. All children busy with a non-zero listen queue means requests are queueing, so continue. Idle children while load is high means the time is going somewhere else: cron, a CLI import, a backup job.

Get timing into the log

The default combined format gives you bytes and user agent but no duration, and duration is what separates a crawler making 200 cheap requests from one making 200 expensive ones.

nginx:

log_format main_ext '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
                    'rt=$request_time';

access_log /var/log/nginx/access.log main_ext;

nginx -t && systemctl reload nginx

Apache: add %D to a custom LogFormat; it's the request duration in microseconds. Existing lines keep the old format, so give it an hour before reading.

Slice by user agent

Both nginx and Apache combined quote the request, the referer and the user agent. That makes " a reliable field separator, and the UA the sixth field:

awk -F'"' '{print $6}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20

You'll get the usual mix: uptime monitors, search engines, your own scripts, and whatever AI crawlers your content attracted. Note the candidates, then move to cost.

Price each bot in worker seconds

Hit counts flatter crawlers that are noisy but cheap. Multiply by time instead:

awk -F'"' '
  {
    split($3, s, " ")        # s[2]=status, s[3]=body bytes
    ua = $6
    rt = $NF; sub(/^rt=/, "", rt)
    hits[ua]++
    bytes[ua] += s[3]
    secs[ua]  += rt
  }
  END {
    for (u in hits)
      printf "%10.1f %8d %13d  %s\n", secs[u], hits[u], bytes[u], u
  }' /var/log/nginx/access.log | sort -rn | head -20

Columns: seconds of request handling, hits, bytes served, user agent. A crawler at 8,000 hits and 40 seconds is background noise. One at 900 hits and 600 seconds is your problem, and it's probably hitting something expensive.

If the file is too large to parse in one pass, sample it:

shuf -n 500000 access.log > sample.log

Ratios survive sampling. Absolute counts don't, so don't quote them anywhere.

Look at the paths the top bot requests

awk -F'"' '$6 ~ /BotName/ {split($2, r, " "); print r[2]}' access.log \
  | sort | uniq -c | sort -rn | head -20

Replace BotName with a token from the previous step. What you're looking for:

• Search results, faceted filters, cart and checkout URLs. Uncacheable, database-backed, and often the entire cost.
• Pagination deep in the archive, or tag pages that nothing links to.
• A single URL requested thousands of times, which is a crawler bug or a redirect loop more than a load problem.

If the expensive paths are the same across several bots, fix the paths first. It's a smaller change and it applies to every visitor.

Prove causality before you throttle

A user agent is a claim, and a throttle is cheap to test. Short-circuit one UA temporarily:

# temporary, inside the server block
if ($http_user_agent ~* "BotName") {
    return 403;
}

Reload, wait 20–30 minutes, and compare load average, PHP-FPM queue depth and database CPU against the same window before the change. If nothing moved, that bot wasn't the cause and you can drop the rule. Then check whether the UA is genuine rather than a scraper wearing it:

dig +short -x 203.0.113.10        # reverse lookup
dig +short crawl.example.com      # forward lookup, should return the same IP

Forward-confirmed reverse DNS is what separates the real crawler from someone running curl with a copied header. It matters most right before you write a permanent rule.

Order to act in

A per-UA rate limit looks like this:

map $http_user_agent $bot_key {
    default          "";
    ~*GPTBot         "gptbot";
    ~*ClaudeBot      "claudebot";
}

limit_req_zone $bot_key zone=ai_bots:10m rate=30r/m;

server {
    limit_req zone=ai_bots burst=20 nodelay;
    limit_req_status 429;
    # ...
}

Requests with an empty key aren't counted, so ordinary visitors are untouched. Use the tokens you actually saw in your own log, not the two above. If your build exposes $limit_req_status, add it to the log format and you can confirm the limits are firing instead of assuming.

robots.txt sits third for a reason: it's advisory. Some crawlers honour it, some don't, and the only way to know which happened is to read the log a day later. Write the file, then verify it, rather than reporting it as done.

Close the loop with the numbers you started with

Run the cost query again over a window after the change. You want the same shape of answer as before: which UA is first, and how many seconds it now costs. If the top entry has changed, you fixed the last problem and this one was already there waiting.

Wrap the whole sequence in a small script and keep it. One command that prints seconds, hits and bytes per user agent, sorted, is worth more than a firewall rule nobody can explain six months from now.

If you'd rather not maintain the parser: AumCrawl is a free plugin that records which AI crawlers are hitting a WordPress site and lets you allow or block them individually from the admin. It sits next to the log work above rather than replacing it — the log is still what tells you what each bot costs.