如何識別 AI 爬蟲而不封鎖 Googlebot

Your website is receiving automated requests. Some are from search engines trying to index your pages. Others come from AI crawlers, monitoring tools, scrapers, or malicious scripts.

The challenge isn't identifying every automated request. It's knowing which bots should be allowed and which ones deserve closer inspection.

Block everything that looks automated, and you risk blocking Googlebot or AI search crawlers. Trust every request claiming to be Googlebot, and anyone can bypass your basic filtering rules.

So how do you detect AI bots without blocking legitimate crawlers?

In this guide, we'll cover the different types of bots, build a working bot detection tool using Node.js, test it in VS Code, and explore how IP intelligence can help classify suspicious traffic.

What you'll build: A local Express server that recognizes bot user-agent strings, plus an optional IP Security lookup script that can be tested with sample data.

The basic project is free. No paid API subscription is required to complete the first part.

1. Understand the Different Types of Bots

Not every bot visiting your website is trying to cause problems.

Here's a quick comparison.

OpenAI, for example, operates different crawlers for different purposes.

According to OpenAI's official documentation, OAI-SearchBot is used for ChatGPT search, while GPTBot crawls content that may be used for training AI models.

ChatGPT-User is different again. It can retrieve web pages in response to a user's action.

The distinction matters because you may want your articles discoverable in AI search without allowing every type of automated crawling.

2. Why User-Agent Detection Isn't Enough

One of the easiest ways to recognize a bot is through its HTTP User-Agent header.

For example, a request might contain:

Googlebot/2.1

Or:

GPTBot/1.4

Your server can inspect that header and classify the request.

But there's a problem.

Anyone can send a request pretending to be Googlebot:

curl -A "Googlebot" https://example.com

The -A option tells curl to send a particular user-agent string.

It doesn't make the request come from Google.

That's why we need to distinguish two concepts:

• Bot identification: What software does the request claim to be?
• Bot verification: Does the source actually belong to the claimed operator?

Let's start with identification.

3. Build a Basic AI Bot Detector in Node.js

This section works locally and doesn't require an IP intelligence API key.

We'll use Node.js, Express, and VS Code.

Step 1: Check your Node.js installation

Download Node.js from nodejs.org if it isn't installed.

Open VS Code and select:

Terminal → New Terminal

Run:

node -v

Then:

npm -v

Both commands should display version numbers.

Use Node.js 20 or newer for this project.

Step 2: Create a project folder

In the VS Code terminal, run these commands:

mkdir bot-detection-demo
cd bot-detection-demo

Initialize the project:

npm init -y

Install Express and dotenv:

npm install express dotenv

Then open your project in VS Code using File → Open Folder and select bot-detection-demo.

Step 3: Create the main server file

In the VS Code Explorer, right-click your project folder and select New File.

Name it:

server.mjs

Paste this code:

import express from "express";

const app = express();

const knownBots = [
  "Googlebot",
  "Bingbot",
  "GPTBot",
  "OAI-SearchBot",
  "ChatGPT-User",
  "ClaudeBot",
  "PerplexityBot"
];

function detectBotClaim(userAgent = "") {
  return knownBots.find(bot =>
    userAgent.toLowerCase().includes(
      bot.toLowerCase()
    )
  ) || null;
}

app.get("/check-bot", (req, res) => {
  const userAgent = req.get("user-agent") || "";

  const detectedBot = detectBotClaim(userAgent);

  res.json({
    message: "Bot identification completed",
    botClaim: detectedBot,
    isClaimingKnownBot: Boolean(detectedBot),
    verified: false,
    note: "User-agent matching does not verify identity."
  });
});

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

Save the file using Ctrl + S.

This code checks incoming requests against a list of common search and AI bot names.

Notice the field:

verified: false

That's intentional. Our code identifies the claimed bot name but doesn't perform any operator verification.

Step 4: Start the server

In your terminal, run:

node server.mjs

You should see:

Server running on port 3000

Your local server is now running.

Open a second VS Code terminal using the + button in the Terminal panel.

Keep the first terminal running.

Step 5: Test Googlebot detection

In the second terminal, run:

curl -A "Googlebot" http://localhost:3000/check-bot

Expected response:

{
  "message": "Bot identification completed",
  "botClaim": "Googlebot",
  "isClaimingKnownBot": true,
  "verified": false,
  "note": "User-agent matching does not verify identity."
}

The server successfully recognized a request claiming to be Googlebot.

Step 6: Test GPTBot detection

Run:

curl -A "GPTBot" http://localhost:3000/check-bot

The response should contain:

{
  "message": "Bot identification completed",
  "botClaim": "GPTBot",
  "isClaimingKnownBot": true,
  "verified": false,
  "note": "User-agent matching does not verify identity."
}

Step 7: Test OAI-SearchBot

Run:

curl -A "OAI-SearchBot" http://localhost:3000/check-bot

The response should identify:

{
  "message": "Bot identification completed",
  "botClaim": "OAI-SearchBot",
  "isClaimingKnownBot": true,
  "verified": false,
  "note": "User-agent matching does not verify identity."
}

Step 8: Test an ordinary request

Now run:

curl http://localhost:3000/check-bot

Since curl doesn't normally claim to be one of the bots in our list, the response should include:

{
  "message": "Bot identification completed",
  "botClaim": null,
  "isClaimingKnownBot": false,
  "verified": false,
  "note": "User-agent matching does not verify identity."
}

That's it.

You've built and tested a working user-agent-based bot identifier.

But it doesn't detect sophisticated automation with unfamiliar user agents, and it doesn't verify whether a recognized name is genuine.

Let's improve the approach.

4. How to Verify Real Googlebot Traffic

Google recommends verifying its crawlers using DNS or published crawler IP ranges.

The DNS verification process works like this:

1. Extract the actual client IP address.
2. Perform a reverse DNS lookup.
3. Check whether the hostname ends with an approved Google crawler domain.
4. Perform a forward DNS lookup on that hostname.
5. Confirm the original IP is among the returned addresses.

For example, Google's documentation shows crawler hostnames under:

googlebot.com

But you shouldn't trust an arbitrary hostname simply because it contains the word google.

The hostname must match Google's documented patterns, and forward DNS must confirm the original address.

Google also publishes official crawler IP ranges.

You can find the complete procedure in the Google crawler verification documentation.

What about OpenAI crawlers?

OpenAI publishes IP ranges for different crawlers:

• OAI-SearchBot IP ranges
• GPTBot IP ranges
• ChatGPT-User IP ranges

If a request claims to be GPTBot, you can compare its actual source IP against the appropriate published ranges.

For production systems, you should regularly refresh these lists and correctly handle IPv4 and IPv6 CIDR ranges.

Also, when your application sits behind a reverse proxy, configure Express's trust proxy setting for your actual trusted proxy infrastructure. Don't blindly trust an incoming X-Forwarded-For header.

5. Detect Unknown Bots Using IP Intelligence

Verifying Googlebot is useful because Google provides official verification methods.

But what about a bot that doesn't identify itself?

Imagine an unknown client makes thousands of requests to your website.

It doesn't claim to be a search crawler. Its user-agent looks ordinary, and you don't recognize the IP address.

In this situation, IP intelligence can provide useful context.

For example, IPGeolocation.io's IP Security API provides information about whether an IP address has been associated with:

• Automated bot activity
• Known legitimate bots
• Brute-force attacks
• VPNs and proxy networks
• Tor exit nodes
• Cloud hosting providers
• Other security risks

The API can return fields such as:

These signals help distinguish a recognized crawler from suspicious automation.

An important limitation is that IP intelligence classifies an address based on available intelligence. It doesn't prove that every individual request from that address came from the same software.

Is the IP Security API free?

The IP Security Lookup feature requires a paid subscription.

According to the current documentation, each successful security lookup costs two API credits. The provider also describes an option to request trial access through customer support.

You can still follow the next example without paid access by using mock security data.

6. Test IP Bot Classification in VS Code

Now we'll create a separate script to inspect security information.

We'll support two testing methods:

• Mock mode: Uses example data locally and requires no paid subscription.
• Live mode: Calls the actual IP Security API when your subscription permits it.

Step 1: Create the test file

Inside the same project, create:

test-ip.mjs

Paste:

import "dotenv/config";
import net from "node:net";

const API_KEY = process.env.IPGEOLOCATION_API_KEY;
const mode = process.argv[2] || "mock";
const ip = process.argv[3] || "4.227.36.0";

function classifySecurity(security) {
  if (security.is_known_attacker) {
    return "Review and consider blocking";
  }

  if (security.is_known_good_bot) {
    return "Verify operator before allowing";
  }

  if (security.is_bot) {
    return "Monitor or challenge";
  }

  return "Allow and monitor";
}

const mockSecurity = {
  is_bot: true,
  bot_type: "ai_crawler",
  bot_operator_name: "Example AI Operator",
  is_known_good_bot: true,
  bot_confidence_score: 95,
  threat_score: 10,
  is_known_attacker: false
};

async function getSecurityData() {
  if (mode === "mock") {
    return mockSecurity;
  }

  if (mode !== "live") {
    throw new Error("Use mock or live mode");
  }

  if (!API_KEY) {
    throw new Error(
      "Add IPGEOLOCATION_API_KEY to your .env file"
    );
  }

  if (!net.isIP(ip)) {
    throw new Error("Invalid IP address");
  }

  const url = new URL(
    "https://api.ipgeolocation.io/v3/security"
  );

  url.searchParams.set("apiKey", API_KEY);
  url.searchParams.set("ip", ip);

  const response = await fetch(url, {
    signal: AbortSignal.timeout(10000)
  });

  if (!response.ok) {
    const error = await response.text();

    throw new Error(
      `API returned ${response.status}: ${error}`
    );
  }

  const data = await response.json();

  if (!data.security) {
    throw new Error("Security object missing");
  }

  return data.security;
}

async function main() {
  try {
    const security = await getSecurityData();

    console.log("\nMode:", mode);

    if (mode === "mock") {
      console.log("Using simulated security data");
    } else {
      console.log("Checked IP:", ip);
    }

    console.log("\nSecurity information:");
    console.log(JSON.stringify(security, null, 2));

    console.log(
      "\nSuggested action:",
      classifySecurity(security)
    );

  } catch (error) {
    console.error("\nTest failed:", error.message);
    process.exitCode = 1;
  }
}

main();

Save the file.

Step 2: Run the free mock test

Open your VS Code terminal and run:

node test-ip.mjs mock

The script should display:

Mode: mock
Using simulated security data

Followed by the sample security information and:

Suggested action: Verify operator before allowing

This verifies your classification logic without making an external API request.

The data is simulated. Don't present this output as real IP intelligence.

Step 3: Configure live API access

If you have access to the paid Security API or an enabled trial, create a file named:

.env

Add:

IPGEOLOCATION_API_KEY=your_actual_api_key

Replace the placeholder with your API key.

Also create .gitignore:

.env
node_modules/

Never publish your real API key in screenshots, code examples, or a GitHub repository.

Step 4: Run a live lookup

Once your account has Security API access, execute:

node test-ip.mjs live 4.227.36.0

This IP appears in IPGeolocation's documentation as an example associated with a recognized AI crawler.

If the request succeeds, you'll receive real security fields and the script's suggested action.

The current classification may differ from the documentation example because IP intelligence changes over time.

Step 5: Understand API access errors

If your account doesn't support the Security API, you may receive an error similar to:

API returned 401:
IP Security Lookup is not supported
on your current subscription.

This doesn't mean your Node.js code is broken.

It means the API received your request but the subscription doesn't permit this feature.

To continue learning, run:

node test-ip.mjs mock

Or check the IP Security API documentation for subscription and trial information.

7. How to Decide Which Bots to Allow or Block

Once you have identification and risk signals, you need a policy.

A sensible starting point looks like this:

Don't automatically block every datacenter IP.

Many legitimate crawlers operate from cloud infrastructure.

Similarly, don't automatically trust a bot simply because its IP reputation looks good.

For sensitive endpoints such as login forms, combine IP intelligence with request frequency, failed-login patterns, rate limits, and other behavioral signals.

The examples in this article return suggested actions. They don't implement firewall rules or automatically block requests.

That's intentional because a production blocking decision requires stronger verification and a tested false-positive policy.

8. Use robots.txt to Manage AI Crawlers

Some crawler operators support separate instructions for different purposes.

For example, you might allow Googlebot and OAI-SearchBot while asking GPTBot not to crawl your website.

Your robots.txt file could include:

User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This tells compliant crawlers that:

• Googlebot may crawl public content.
• OAI-SearchBot may crawl public content.
• GPTBot should not crawl the site.

Remember that robots.txt is not a security mechanism.

Malicious bots can ignore it.

OpenAI also notes that user-triggered ChatGPT-User requests are different from its automatic crawlers, and ordinary robots.txt rules may not apply in the same way.

When you need actual enforcement, use verified network identity and appropriate server-side access controls.

9. Common Bot Detection Mistakes

Trusting user-agent strings

Anyone can claim to be Googlebot.

A matching string isn't proof of identity.

Blocking every cloud IP

Legitimate crawlers, monitoring tools, and real users may access your website from cloud infrastructure.

Treating all AI bots identically

AI search crawlers, training crawlers, and assistant-triggered fetchers serve different purposes.

Making an IP lookup for every request

External API calls introduce latency and cost.

Cache appropriate results and avoid unnecessary repeated lookups.

Blocking based on a single risk indicator

A flagged IP address doesn't necessarily prove malicious intent for every request.

Use additional signals and provide a way to investigate false positives.

Ignoring API failures

Your website shouldn't become unavailable just because a security intelligence API temporarily fails or your account reaches its limits.

Choose an explicit failure policy. For public informational pages, continuing with monitoring may be reasonable. Sensitive endpoints may require stricter controls.

10. Frequently Asked Questions

How do I detect AI bots on my website?

Start by examining user-agent strings and request logs. Verify recognized crawlers using their operators' published IP ranges or DNS verification methods. For unknown traffic, combine behavioral analysis with IP intelligence.

Can someone fake Googlebot?

Yes. User-agent headers can be changed freely. Google recommends verifying crawler requests using its documented DNS procedure or official IP ranges.

Can I test bot detection without paying for an API?

Yes. The basic Node.js user-agent detector in this article runs locally without an API key. You can also test the classification logic using simulated security data. Real IP Security lookup results require the appropriate API subscription or trial access.

Will blocking AI training crawlers affect Google rankings?

Blocking GPTBot through its specific robots.txt rule doesn't directly block Googlebot. However, broad server or firewall restrictions could accidentally affect search engine access.

Is IP intelligence enough to identify malicious bots?

No. It's one useful layer. Combine it with identity verification, request rates, behavioral patterns, and your website's own access policy.

Final Thoughts

Detecting AI bots isn't simply about checking whether a request contains the word bot.

You need to understand the traffic, identify known crawlers, verify their identities when necessary, and make decisions based on both reputation and behavior.

In this guide, we built a basic bot identifier, tested Googlebot and AI crawler user-agent strings, and added an optional IP Security classification example.

The most important lesson is simple:

Don't block a request just because it looks automated, and don't trust a request just because it claims to be legitimate.

Verify what you can, monitor what you can't, and apply policies that protect your website without unnecessarily restricting search engine access.

How are you handling AI crawlers on your own websites? Are you allowing them, blocking certain types, or monitoring everything first?