跳到正文
DEV Community· ranaiman·· 2026-10-15AI 評分42

Which AI crawlers do 261 top websites block? Study data

Which AI crawlers do 261 top websites block? Study data

AI 導讀

對 261 個熱門網站的 robots.txt 檔案進行分析後,研究發現 35% 至少封鎖一種訓練爬蟲,21% 至少封鎖一種搜尋爬蟲。CCBot 和 Bytespider 各有 28.7% 的網站封鎖,是最常被封鎖的爬蟲;OAI-SearchBot 的封鎖率最低,為 11.5%。在 48 個新聞網站中,有 27 個封鎖 GPTBot。

正文

AI Ranked profile image ranaiman

We checked the robots.txt files of 285 of the most visited websites to see how they treat AI crawlers. 261 of them returned a readable file. This post sums up what we found. The full report and the raw data are linked at the end.

The headline numbers

  • 59 of the 261 sites (22.6%) disallow GPTBot, the crawler OpenAI uses to collect training data.
  • 35% of sites block at least one training crawler.
  • 21% of sites block at least one search crawler, the bots that fetch pages for live answers rather than for model training.
  • CCBot (Common Crawl) and Bytespider (ByteDance) are the most blocked user agents, at 28.7% each.
  • OAI-SearchBot, the crawler behind ChatGPT search results, is the least blocked at 11.5%.
  • Only one site in the whole set blocks Googlebot: Reddit.

News sites drive most of the blocking

The sample includes 48 news sites, and 27 of them block GPTBot. That is nearly half of all GPTBot blockers in the study coming from a single category. Publishers have clear reasons to keep training crawlers out, and the numbers show they act on it far more than retailers, software companies or reference sites do.

Training crawlers vs search crawlers

The gap between 35% and 21% matters. Many sites that shut out training bots still let search bots in, because being fetched for a live answer in ChatGPT, Perplexity or Google AI Overviews can send visitors, while training access does not. OAI-SearchBot being the least blocked agent fits that pattern. Bytespider sits at the other end: it is blocked almost as often as CCBot even though it is far less known, likely because it has a reputation for ignoring rules.

Get the data

The CSV with every site, every crawler and every rule is open on GitHub: ai-crawler-access-study

The full write up, including method and per category breakdowns, is here: AI crawler access report

If you want to check your own robots.txt against these crawlers, AI Ranked has a free in-browser robots.txt checker.

來源:DEV Community · dev.to