如何為網頁抓取的 RAG 內容進行結構感知分塊

A chunk can fit your token limit and still be a bad retrieval result: half a code sample, a table row without its header, or an answer separated from its heading.

For crawled documentation, try structure-aware splitting first, then apply a tokenizer limit to oversized sections.

Three useful baselines

• Fixed windows are predictable and make a good baseline. They may cut through sentences and code.
• Recursive splitting prefers paragraph and sentence boundaries, then falls back to smaller units.
• Heading-aware splitting preserves document sections and lets you attach the heading path as metadata.

This standard-library example keeps fenced code together while producing sections:

FENCE = chr(96) * 3

def heading_chunks(markdown: str):
    path, lines, fenced = [], [], False

    for line in markdown.splitlines():
        if line.strip().startswith(FENCE):
            fenced = not fenced

        if line.startswith("#") and not fenced and lines:
            body = "\n".join(lines).strip()
            if body:
                yield " / ".join(path), body

            level = len(line) - len(line.lstrip("#"))
            path = path[:level - 1] + [line[level:].strip()]
            lines = [line]
        else:
            lines.append(line)

    if lines:
        body = "\n".join(lines).strip()
        if body:
            yield " / ".join(path), body

chunks = [
    {"heading_path": path, "text": text}
    for path, text in heading_chunks(markdown)
]

It is a first pass. A single large section can still exceed the embedding model’s input limit, so split that section again using paragraphs and a tokenizer.

Keep hierarchy with each chunk

If a child section is called “Limits,” the parent path may explain whether it means API limits, model limits, or account limits. Store that path as metadata instead of duplicating the full page into every vector.

Treat special elements deliberately:

• Keep a table with its header row.
• Keep a parameter description close to the parameter name.
• Keep short code examples with their imports where possible.
• Preserve FAQ question and answer pairs.
• Keep source URL, page version or crawl time, and content hash.

Compare on one fixed evaluation set

Do not pick a strategy by looking at pretty boundaries alone. Create 10–30 queries from real documentation tasks. Include parameter lookup, a cross-section question, a table lookup, a code question, and an unanswerable query.

For each strategy, inspect the top results and record whether the answer-bearing text appears in the top k, whether heading and source metadata are present, whether code or table context was lost, whether duplicate chunks appear, and how much prompt token cost changes.

Use the same pages, embedding model, index settings, and questions for each run. The LangChain text-splitter guide explains common recursive options; the OpenAI embeddings guide describes embedding inputs and model behavior.

An evaluation table can start with five columns: question, expected source section, retrieved top-k sections, answerable from context, and notes. Review failures by category. If parameter questions fail but table lookups pass, changing global chunk size may make the working cases worse; try preserving parameter blocks or heading context first.

A limit such as 1,200 characters is not 1,200 tokens. Use your actual tokenizer to enforce the model’s input ceiling. Keep overlap small and add it only when evaluation shows that a boundary removed context. Large overlap can raise duplicate matches and storage without improving answers.

No strategy wins for every corpus. Choose the simplest one that retrieves complete, citable evidence on your own test questions.