哪些博客平台允許 AI 搜尋爬蟲讀取文章？Robin 檢查 robots.txt 與原始 HTML（2026-10-31）
Which blogging platforms let AI search bots read your posts? (robots.txt and raw HTML, checked 31 Oct 2026)
Robin 檢查多個博客平台的 robots.txt、原始 HTML 和標題搜尋結果，發現 dev.to、Substack 和 Gumroad 的 robots.txt 沒有禁止 GPTBot、OAI-SearchBot、PerplexityBot 和 Google-Extended；Medium 只禁止 GPTBot，Indie Hackers 則禁止全部四種。
Posted on Nov 2 AI-assisted
Short answer: in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.
Small sample: one day, one machine, plain curl with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.
robots.txt, per host
| host | GPTBot | OAI-SearchBot | PerplexityBot | Google-Extended |
|---|---|---|---|---|
| dev.to | allowed | allowed | allowed | allowed |
| aicitationnotes.substack.com | allowed | allowed | allowed | allowed |
| normbrytande.gumroad.com | allowed | allowed | allowed | allowed |
| medium.com | disallowed (/) |
allowed (*) |
allowed (*) |
allowed (*) |
| indiehackers.com | disallowed (/) |
disallowed (/) |
disallowed (/) |
disallowed (/) |
Notes:
- "Allowed" means robots.txt does not forbid it. Whether a CDN or WAF blocks the real bots is not verified.
- GPTBot is OpenAI's training crawler. OAI-SearchBot is the one used for ChatGPT search. Medium only blocks the first.
- dev.to serves a
/llms.txt(200). It asks that AI-assisted articles setai_disclosure_levelaccurately.
Is the article text in raw HTML?
curl -sL --compressed -A "<Chrome desktop UA>" "$URL"
- dev.to: all 5 of my posts, full text present, 692 to 1,657 words each.
- Substack and Indie Hackers: full text present.
- Medium: 403 Cloudflare challenge on profile and all 6 posts. The RSS feed returned full content.
- Gumroad product pages: body text only inside an Inertia
data-pageJSON attribute, plus<title>, meta description and Product JSON-LD.
Exact-title lookups
With a web search tool (backend not disclosed, so not tied to any specific engine): all 5 of my dev.to posts were found by exact title. None of my 6 Medium posts were found, and for three cross-posted ones the dev.to copy came up instead.
What the two dev.to posts Perplexity cited had in common
For "How do I check if ChatGPT recommends my product?", Perplexity cited two dev.to posts by other authors, both with 0 reactions. One has the question as its title and answers in the first sentence. The other has a near-question title and answers within 3 short paragraphs, then lists numbered steps. My posts open with a story. Two examples are not proof, so the next test is one page with the question as H1 and the answer first.
來源：DEV Community · dev.to