跳到正文
DEV Community· Muhammad Ahmad·· 11 天前AI 評分49

如何在 Next.js App Router 網站加入 llms.txt、llms-full.txt 和 W3C tdmrep.json

How to Add llms.txt, llms-full.txt, and W3C tdmrep.json to a Next.js App Router Site

AI 導讀

教學説明如何在 Next.js App Router 網站設定 llms.txt、llms-full.txt 和 W3C tdmrep.json,讓 AI 編碼助手和搜尋引擎讀取機器可讀資訊。內容涵蓋 llms.txt 的 Markdown 格式要求、透過靜態路由處理器提供檔案,以及 RFC 9309 robots.txt 常見錯誤和 Cloudflare 規則衝突。

正文

If you maintain developer documentation, an API platform, or a SaaS web app in Next.js (App Router), adding machine-readable endpoints for AI coding assistants (Cursor, Windsurf, GitHub Copilot, Claude Code) and AI search engines (OAI-SearchBot, PerplexityBot) takes less than 10 minutes.

In this tutorial, we will look at the exact structure required by the llmstxt.org specification, the 3 most common RFC 9309 robots.txt mistakes that break AI crawlers, and how to serve /llms.txt and /.well-known/tdmrep.json using Next.js static route handlers.


1. The Strict llmstxt.org Markdown Grammar

Unlike sitemap.xml, /llms.txt is designed to be read by both deterministic parsers and LLM tokenizers. The specification enforces a strict ordering:

# Project or Platform Name

> A concise, factual 1-to-2 sentence summary of what the platform does, its core architecture, and target audience (zero marketing superlatives).

Optional concise operational notes (e.g., authentication model, rate limits, or API base URL).

## Core Product & Tools

- [JSON Formatter & Validator](https://example.com/developer/json-formatter): Strict RFC 8259 JSON validator with 64-bit BigInt truncation detection and JSONPath evaluation.
- [HAR File Analyzer & Sanitizer](https://example.com/developer/har-analyzer): Client-side 7-phase HTTP waterfall inspector and 14-surface secret scrubber.

## Optional

- [Privacy Policy](https://example.com/privacy): Details on client-side data handling and GDPR/CCPA compliance.

Three Linter Rules to Watch Out For:

  1. Exactly One # H1 at Line 1: Parsers expect the document to begin with a single H1 heading followed immediately by the > summary blockquote.
  2. No Vague Link Anchors: Never write - [Click here](...) or - [Read more](...). The anchor text and colon-separated description are what the LLM uses to decide whether to fetch the underlying URL.
  3. Use the ## Optional Section for Secondary Pages: According to the spec, links under ## Optional tell context-constrained models which URLs can be skipped when token budgets are tight.

2. Serving /llms.txt in Next.js App Router (app/llms.txt/route.ts)

In Next.js App Router, you can serve /llms.txt at the edge with zero runtime overhead using force-static and Cache-Control headers:

// app/llms.txt/route.ts
export const dynamic = 'force-static';

const LLMS_TXT_CONTENT = `# YourPlatformName

> Factual summary of your platform and technical architecture.

## Docs & Core Tools
- [API Reference](https://example.com/docs/api): Complete REST & GraphQL endpoint specifications.
`;

export async function GET() {
  return new Response(LLMS_TXT_CONTENT, {
    status: 200,
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=3600, s-maxage=86400, stale-while-revalidate=604800',
      'X-Robots-Tag': 'all',
    },
  });
}

3. Auditing Your robots.txt for RFC 9309 Prefix Leaks

Under IETF RFC 9309, Disallow: /admin matches /admin, /admin/, AND /admin-guide.html (because Disallow is a prefix match!). Conversely, if you write Disallow: /private/, a request to /private (without the trailing slash) is still allowed.

And if you use Cloudflare in front of Next.js, verify that Cloudflare's Managed robots.txt ("Block AI training bots") isn't prepending its own rules above your app/robots.ts output—which frequently causes inconsistent AI bot policies across Amazonbot, Claude-Web, and Meta-ExternalAgent.

If you want to auto-convert your existing sitemap.xml or HTML docs into llms.txt, llms-full.txt, W3C tdmrep.json, and Next.js route code—while profiling token usage across GPT-4o (128k), Claude 3.7 Sonnet (200k), and DeepSeek R1 (64k)—you can use the client-side KivaroTools llms.txt & AI Crawler Generator.

來源:DEV Community · dev.to