跳到正文
DEV Community· Віталік Бакута·· 7 天前AI 評分44

你的網站可能對 ChatGPT 隱形,而你未必會察覺

Your website might be invisible to ChatGPT (and you'd never know)

AI 導讀

作者發現其網站雖在 Google 排名正常,ChatGPT、Claude 和 Perplexity 卻沒有提及;robots.txt 阻擋 AI 爬蟲、內容依賴 JavaScript,都可能令網站難以被 AI 系統讀取。

正文

Віталік Бакута

Last month I noticed something weird: my site ranked fine on Google, but when I asked ChatGPT, Claude and Perplexity about my topic, they never mentioned it. Not once.

So I dug in. Turns out there's a whole layer of "AI visibility" most of us never check.

What I found

1. My robots.txt was blocking AI crawlers I didn't even know existed.
There are 20+ AI bots now: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more. Some hosting setups and security plugins block them by default. Cloudflare can too.

2. My content was rendered with JavaScript.
Many AI crawlers don't execute JS the way Googlebot does. If your content only appears after hydration, some bots see an empty page.

3. No structured data.
Schema markup helps AI systems understand what your page is: a product, an article, a tool.

How to check your own site (2 minutes)

I ended up building a set of free checkers for this: BotReady. A few worth trying:

5 free checks a day, no sign-up.

Quick fixes if you're blocked

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

(Only if you want AI tools to use your content. That's a real choice, not a default.)

Your turn

Run your site through a checker and drop your result in the comments. I'm curious how many of us are accidentally invisible. 👇


Side projects of mine: FormatMath (file converters that run in your browser) and Fee Math (seller fee calculators).

來源:DEV Community · dev.to