如何為任何網站加入 llms.txt 及 AI 搜尋爬蟲規則(OAI-SearchBot、PerplexityBot、ClaudeBot)
How to Add llms.txt and AI Search Bot Rules (OAI-SearchBot, PerplexityBot, ClaudeBot) to Any Website
網站可在根目錄放置 Markdown 格式的 llms.txt,協助 AI 助手及 coding agents 閲讀核心文件,並以 robots.txt 的不同 user-agent 規則分開管理 AI 搜尋引用與模型預訓練存取。範例允許 OAI-SearchBot、PerplexityBot 和 Claude-SearchBot 瀏覽,並禁止 GPTBot 存取。
Traditional sitemap.xml files list every URL on your domain for search engine crawlers, whereas llms.txt is a lightweight Markdown map placed at your site root (yourdomain.com/llms.txt) that helps AI assistants and coding agents read your core documentation without HTML navigation clutter.
1. Separating AI Search Citations from Model Training in robots.txt
If you want your technical guides cited in ChatGPT Search, Perplexity, or Claude answers while controlling model pre-training access, split your user-agent rules:
text
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
來源:DEV Community · dev.to