Cloudflare 協助 ChatGPT 發現新網頁，作者整理避免爬蟲略過頁面的 5 項檢查

On October 15, Cloudflare and OpenAI shared early results from a research pilot they started in July. Cloudflare passed signals from its network to OpenAI's search crawler, and according to Cloudflare's write-up, that helped OpenAI discover and index "hundreds of millions of new web pages" for ChatGPT search. The pilot only covered about 1% of Cloudflare's network.

The same day, Google announced changes to AI Mode and AI Overviews: more inline links shown higher in responses, and a test of a section at the top of some answers that surfaces the one or two "official places on the web" to complete a task.

Put those together and the message for developers is pretty plain. AI search is getting better at finding pages, and it's getting pickier about which ones are worth a visit. The interesting part of the Cloudflare post isn't the headline number. It's the list of things that make crawlers waste time, because most of them are bugs we ship ourselves.

Here's what I'd check on my own site this weekend, plus a small Node script that automates most of it.

What the pilot actually says (and doesn't)

It's worth being precise, because "Cloudflare now feeds your site to ChatGPT" is not what happened.

• Cloudflare provides signals that help OpenAI's crawler find pages. OpenAI then "independently evaluates and ranks" them.
• The post says the research uses metadata about the web for search and discovery, "rather than sharing page content for model training."
• "A signal about a page doesn't grant permission to access it." Your robots.txt and your Cloudflare settings still decide access.
• The pilot was a ~1% cohort. There's no setting to opt into it, and the post doesn't describe one.

So there's nothing to toggle. What you can control is whether your site sends clean signals when a crawler does show up.

1. Know which OpenAI bot does what

OpenAI documents three main agents, and they're independent:

• OAI-SearchBot surfaces sites in ChatGPT's search features. If you block it, OpenAI says your site won't be shown in ChatGPT search answers, though it can still appear as a navigational link.
• GPTBot crawls content that may be used to train OpenAI's foundation models. Blocking it is the training opt-out.
• ChatGPT-User fetches pages when a user asks ChatGPT something. OpenAI notes robots.txt rules "may not apply" here because a person started the request, and it isn't used to decide what appears in search.

A lot of sites blocked "everything AI" in 2024 with a single rule and never looked again. If you want to show up in ChatGPT search but stay out of training, the split looks like this:

User-agent: OAI-SearchBot
Allow: /
Disallow: /dashboard
Disallow: /settings

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

OpenAI says changes can take about 24 hours to be picked up for search. If you're on Cloudflare, AI Crawl Control (available on all plans) shows which AI crawlers actually hit your site and whether they follow your robots.txt, which is a better source of truth than what you think your file says.

2. Make missing pages return a real 404

This is the one the Cloudflare post calls out most directly. They found that "bots repeatedly ask for files that aren't there," that "some sites redirect almost any address to their homepage," and that other addresses "lead to copies of the same content." Recognizing those patterns lets them skip fetches entirely.

If your site answers /anything-at-all with a 200 and your homepage, or a 302 to /, every crawler has to fetch and compare to figure out the page doesn't exist. Google's docs call an error page served with a 2xx status a soft 404. Google also notes that 4xx URLs are dropped from its index over time and that "newly encountered 404 pages aren't processed." That's exactly what you want for junk URLs.

The usual culprit is an SPA fallback rewrite, where every path serves index.html with 200. In a Next.js App Router project, the fix for dynamic routes is to call notFound() when the data doesn't exist, which renders your not-found page with a 404 status:

// app/blog/[slug]/page.tsx
import { notFound } from "next/navigation";
import { getPost } from "@/lib/posts";

export default async function Page({ params }: { params: Promise<{ slug: string }> }) {
  const { slug } = await params;
  const post = await getPost(slug);
  if (!post) notFound(); // 404 status instead of an empty 200 page
  return <article>{post.title}</article>;
}

If a page is gone for good, 410 is fine too. Google treats all 4xx codes except 429 the same way.

3. Collapse your URL variants into one

"Copies of the same content" usually means the same page lives at several addresses: http and https, www and bare domain, with and without a trailing slash, with tracking parameters. Pick one version and permanently redirect (301 or 308) everything else to it. Google treats 301 as a strong signal and 302 as a weak one, so a temporary redirect for a permanent move is a quiet mistake.

Then make sure your <link rel="canonical">, your sitemap URLs, and your internal links all use that same version. Google's sitemap docs say to list only the canonical URLs, as absolute URLs.

4. Send change signals that are true

Cloudflare says the next phase will test "faster and more reliable change signals," because "a crawler that repeatedly downloads an unchanged page adds load without bringing any benefit." Two signals already exist, and you probably have one of them half-working.

Sitemap lastmod. Google says it uses lastmod only "if it's consistently and verifiably" accurate, and that it should reflect the last significant update (content, structured data, or links), not a copyright year. The common bug is a generator that stamps every URL with the build time. After a while, crawlers learn that your lastmod means nothing.

ETag / Last-Modified and 304. When a crawler sends If-None-Match or If-Modified-Since and the page hasn't changed, a 304 Not Modified saves both sides the full download. Google documents that a 304 tells its pipeline the content is the same as last time. Many framework and CDN setups strip these headers or always return 200, so it's worth testing instead of assuming.

5. Look for the one compromised page

The part of the post I didn't expect: Cloudflare and OpenAI are testing signals that flag potentially harmful pages individually. Their example is "a fake login form planted by an attacker" on an otherwise legitimate site. Flagging just that page keeps people safe "without penalizing the site owner."

That's good news, but you'd still rather find the page first. A few things I'd check:

• Pages you didn't create, especially under upload folders or old CMS paths.
• Forms whose action posts to a domain you don't own.
• Old subdomains still pointing at services you no longer use (an easy takeover target).
• Missing security headers like Content-Security-Policy with a form-action rule, which limits where forms on your pages can submit.

A five-minute script for checks 1 to 4

This Node script (Node 18+, no dependencies) checks your robots.txt rules for OAI-SearchBot and GPTBot, whether a random URL returns a real 404, whether host variants permanently redirect, whether your sitemap has believable lastmod values, and whether a page answers a conditional request with 304.

// crawl-check.mjs: run with `node crawl-check.mjs https://example.com`
// Needs Node 18+ (built-in fetch). No dependencies.
const origin = new URL(process.argv[2] ?? "https://example.com").origin;
const UA = "crawl-check/1.0 (+self-audit)";
const get = (url, opts = {}) =>
  fetch(url, { redirect: "manual", ...opts, headers: { "user-agent": UA, ...opts.headers } });
const ok = (msg) => console.log(`  OK   ${msg}`);
const warn = (msg) => console.log(`  WARN ${msg}`);

// 1. robots.txt: which rules apply to OpenAI's search and training bots?
async function checkRobots() {
  console.log("\n1. robots.txt");
  const res = await get(`${origin}/robots.txt`);
  if (res.status !== 200) return warn(`robots.txt returned ${res.status}`), [];
  const lines = (await res.text()).split(/\r?\n/).map((l) => l.replace(/#.*/, "").trim());
  const groups = []; // [{ agents: [], rules: [] }]
  let current = null;
  const sitemaps = [];
  for (const line of lines) {
    const [rawKey, ...rest] = line.split(":");
    const key = rawKey?.toLowerCase();
    const value = rest.join(":").trim();
    if (key === "sitemap") { sitemaps.push(value); continue; }
    if (key === "user-agent") {
      if (!current || current.rules.length) groups.push((current = { agents: [], rules: [] }));
      current.agents.push(value.toLowerCase());
    } else if ((key === "allow" || key === "disallow") && current) {
      current.rules.push(`${key}: ${value || "(empty)"}`);
    }
  }
  for (const bot of ["oai-searchbot", "gptbot"]) {
    const group = groups.find((g) => g.agents.includes(bot)) ?? groups.find((g) => g.agents.includes("*"));
    const label = group?.agents.includes(bot) ? bot : `${bot} (falls back to *)`;
    const blocksAll = group?.rules.includes("disallow: /");
    (blocksAll ? warn : ok)(`${label}: ${group ? group.rules.join(", ") || "no rules" : "no group, allowed"}`);
  }
  return sitemaps;
}

// 2. Does a URL that can't exist return a real 404/410?
async function checkMissingPage() {
  console.log("\n2. Missing pages");
  const res = await get(`${origin}/this-page-should-not-exist-${Date.now()}`);
  if (res.status === 404 || res.status === 410) return ok(`random URL returned ${res.status}`);
  const where = res.headers.get("location");
  warn(`random URL returned ${res.status}${where ? ` -> ${where}` : ""} (soft 404 risk)`);
}

// 3. Do the protocol and www variants collapse into one host with a permanent redirect?
async function checkHostVariants() {
  console.log("\n3. Host variants");
  const { hostname } = new URL(origin);
  const bare = hostname.replace(/^www\./, "");
  const variants = [`http://${bare}/`, `http://www.${bare}/`, `https://${bare}/`, `https://www.${bare}/`];
  for (const v of variants) {
    if (v === `${origin}/`) continue;
    try {
      const res = await get(v);
      const loc = res.headers.get("location");
      if ([301, 308].includes(res.status)) ok(`${v} -> ${res.status} ${loc}`);
      else warn(`${v} -> ${res.status}${loc ? ` ${loc}` : ""}`);
    } catch (err) {
      console.log(`  --   ${v} failed: ${err.cause?.code ?? err.message} (fine if you never use it)`);
    }
  }
}

// 4. Does the sitemap carry lastmod values that look real?
async function checkSitemap(sitemaps) {
  console.log("\n4. Sitemap lastmod");
  const url = sitemaps[0] ?? `${origin}/sitemap.xml`;
  const res = await get(url);
  if (res.status !== 200) return warn(`${url} returned ${res.status}`);
  const xml = await res.text();
  const locs = [...xml.matchAll(/<loc>\s*(.*?)\s*<\/loc>/g)].map((m) => m[1]);
  const mods = [...xml.matchAll(/<lastmod>\s*(.*?)\s*<\/lastmod>/g)].map((m) => m[1]);
  console.log(`  --   ${url}: ${locs.length} <loc>, ${mods.length} <lastmod>`);
  if (xml.includes("<sitemapindex")) console.log("  --   this is a sitemap index; run the check on a child sitemap too");
  if (!mods.length) return warn("no lastmod values, so crawlers get no change signal");
  const unique = new Set(mods.map((m) => m.slice(0, 10)));
  if (mods.length > 5 && unique.size === 1) warn(`every lastmod is ${[...unique][0]} (looks like build time, not real edits)`);
  else ok(`${unique.size} distinct lastmod dates`);
  return locs.find((l) => l.startsWith(origin) && !l.endsWith(".xml"));
}

// 5. Does an unchanged page answer a conditional request with 304?
async function checkConditional(pageUrl = `${origin}/`) {
  console.log("\n5. Conditional requests");
  const first = await get(pageUrl);
  const etag = first.headers.get("etag");
  const lastModified = first.headers.get("last-modified");
  if (!etag && !lastModified) return warn(`${pageUrl} sends neither ETag nor Last-Modified`);
  const headers = {};
  if (etag) headers["if-none-match"] = etag;
  if (lastModified) headers["if-modified-since"] = lastModified;
  const second = await get(pageUrl, { headers });
  (second.status === 304 ? ok : warn)(
    `${pageUrl}: ETag ${etag ? "yes" : "no"}, Last-Modified ${lastModified ? "yes" : "no"}, revalidation -> ${second.status}`
  );
}

console.log(`Checking ${origin}`);
const sitemaps = await checkRobots();
await checkMissingPage();
await checkHostVariants();
const samplePage = await checkSitemap(sitemaps);
await checkConditional(samplePage);

Run it with node crawl-check.mjs https://your-site.com. It doesn't follow redirects on purpose, so you see exactly what a crawler sees on the first hop. If your sitemap is an index, point the check at one of the child sitemaps too.

When I ran it against a couple of well-run sites, even they had a warning or two (usually missing validators or a redirect on the conditional request). That's normal. The goal is to know which ones are deliberate.

What I'd do first

If you only have an hour:

1. Decide your policy for OAI-SearchBot and GPTBot separately, and write it down in robots.txt.
2. Hit a made-up URL on your site and make sure it returns 404.
3. Fix any 302 that should be a 301.
4. Stop stamping build time into lastmod.

None of this is new SEO advice. What changed this week is that two of the biggest AI search systems said, in public, that these exact patterns decide what gets discovered and how much crawling your site attracts.

If you'd rather get the list without writing scripts, I built Siteory for this: it runs a free audit of a site's SEO, AI-search readiness, and security, and gives you prioritized, developer-ready fixes. Either way, the made-up-URL test takes ten seconds. I'd start there.