RAG 文件抓取實測：4 個網站的單頁佔該站 token 總量 39% 至 69%

If you build a RAG bot on top of someone's documentation, the first crawl decides what your retriever will see. I wanted to know what a naive crawl of popular docs sites actually returns, so on 3 November 2026 I pointed a crawler at six of them and counted.

Disclosure: I built the crawler used here, Website to LLM Markdown on Apify. It turns pages into clean Markdown and token-counted chunks. Setup is at the end.

The setup

• Sites: docs.n8n.io, fastapi.tiangolo.com, docs.stripe.com, supabase.com/docs, docs.astro.build, react.dev/learn
• 25 pages per site, breadth-first from the start URL, same domain only, robots.txt respected
• Boilerplate (navigation, footers, cookie banners) removed, then split into chunks of at most 500 tokens along headings
• Tokens counted with the o200k_base tokenizer

All 150 pages came back fine. Together they are 394,230 tokens in 1,037 chunks.

Result 1: one page dominates

On four of the six sites, a single page holds between 39% and 69% of all tokens in the crawl. In three of those cases it is the kind of page you rarely want in a support bot:

• FastAPI's release notes: 117,704 tokens, 287 chunks. One version history page outweighs the other 24 pages combined.
• Stripe's testing page: 36,764 tokens, mostly long tables of test card numbers and error codes.
• n8n's changelog: 13,522 tokens, 37 chunks.

React's big page is a step-by-step tutorial. That one belongs in the index, but it is 53 chunks of very similar code, which can crowd out other answers at retrieval time.

The point is not that these sites are badly built. Changelogs and reference tables are useful to humans. In a vector index they mostly add near-duplicate chunks that match many queries a little and answer none.

Result 2: translations eat the crawl budget

FastAPI's docs are translated into many languages, and the home page links to each version. A breadth-first crawl therefore spends its first steps on translations:

• Of the first 25 FastAPI pages, 12 were translated copies of the home page: Hindi, French, German, Spanish, Korean, Japanese, Turkish, Russian, Portuguese, Chinese (simplified and traditional) and Ukrainian.
• Only 13 pages were English, and only a handful were actual tutorial content.

If your bot answers in English, those pages are pure noise. And the same content in 12 languages can make a retriever return the Japanese version of the answer to an English question.

Result 3: tokens per word vary 2x

Tokens per word is a quick way to see how "dense" a site's Markdown is:

Plain English prose sits around 1.3. Stripe is more than twice that because its pages are full of IDs, parameter tables and code. FastAPI is pushed up by code samples and by the non-Latin scripts of the translated pages.

For budgeting, estimating tokens from word count can be off by a factor of two depending on the site. Count tokens with the tokenizer you actually use.

Three filters to set before indexing a docs site

Based on this, I now set these on every docs crawl:

1. Exclude changelogs and release notes: patterns like */changelog*, */release-notes*, */releases*. If users ask "what changed in version X", index only the last few versions as a separate collection.
2. Pin the language: include only /en/ paths, or exclude the language prefixes (/de/, /ja/, ...). Also check the language field of each page after the crawl.
3. Cap page size: anything over ~10,000 tokens deserves a look before it goes in. It is usually a changelog, a giant reference table or an auto-generated API page.

With those three filters, the FastAPI crawl would have dropped 12 translations and the release notes: 13 of 25 pages and well over half of its tokens.

How to reproduce this

Each run below took 6 to 14 seconds for 25 pages and used well under a cent of compute:

{
  "startUrls": [{ "url": "https://fastapi.tiangolo.com/" }],
  "maxPages": 25,
  "chunkSize": 500,
  "excludeUrlPatterns": ["*/release-notes*", "*/changelog*", "*/hi/*", "*/ja/*", "*/zh/*"]
}

The actor returns one item per page with markdown, wordCount, tokenCountEstimate, language and a chunks array. Each chunk has headingPath, text, tokenCount and a stable chunkId, so you can upsert straight into a vector store. Pricing is $0.0015 per page converted; failed, empty and robots-blocked pages are free.

The analysis above is a few lines of Python on the dataset:

from collections import Counter
pages = [p for p in rows if p["status"] == "ok"]
total = sum(p["tokenCountEstimate"] for p in pages)
biggest = max(pages, key=lambda p: p["tokenCountEstimate"])
print(total, biggest["url"], round(biggest["tokenCountEstimate"] / total, 2))
print(Counter(p["language"] for p in pages))

Run it once before you index anything. Five minutes of looking at where the tokens go saves a lot of debugging later.

Data: 6 public docs sites, 25 pages each, crawled 3 November 2026. A different start URL or page limit gives different pages, so check your own crawl the same way.