如何為 RAG 語料去重而不遺失引用來源

Your embedding deduplication can work perfectly while your citation index quietly forgets a source.

If your ingestion loop keeps a single page row per contentHash, repeated content can replace an earlier source relationship. The vector index looks tidy. The bibliography has selective memory.

Lab note 042: our hypothesis is that embedding content and citation provenance need separate identities. The catch: contentHash has no specified hashing scope, so it can't safely stand for the exact text sent to your embedding model.

Which URL should own a citation?

Deduplicating embedding text must not delete the source observations that support its citations.

For your documentation corpus, the task is narrower than “find the real URL.” You need to retain where each record came from, identify the exact text sent for embedding, and reconnect retrieved text to its sources.

The relationship should look like this:

source observation ─┐
                    ├── exact embedding text ── embedding
source observation ─┘

The Markdown Actor is a useful worked example because it exposes several address variants alongside content and collection metadata.

Those addresses answer different questions. None deserves promotion to a universal document key just because it looks authoritative.

Map fields to jobs before choosing keys

This mapping separates what the retriever needs from what the citation and ingestion layers need.

Fields	Job	Our handling
`url`	Page address	Use the exact string as the local source key; retain it on every observation.
`loadedUrl`, `redirectedUrl`	Fetch destination; final address when different from `url`	Preserve both when present; consider them for citation selection.
`canonicalUrl`	Page-supplied preferred address	Retain as a hint, without assuming uniqueness or equivalence.
`markdown`	Selected page content	Use as the exact embedding payload in this example.
`contentHash`	Actor SHA-256 change signal	Keep for observation comparisons, separate from the application content key.
`scrapedAt`	Collection time	Keep with the observation, outside the embedding payload.
`title`, `author`, `publishedDate`	Identification and attribution	Attach available values to citations, outside the content key.
`httpStatusCode`, `fetchError`	Response and failure context	Retain for auditing; apply an explicit indexing policy.

Page-supplied metadata is conditional. Fields the page doesn't provide are omitted rather than emitted empty.

That matters for chronology: an absent publishedDate stays unknown. Substituting scrapedAt would turn “collected recently” into “published recently.” A very efficient way to manufacture freshness.

Keep provenance outside the embedding

Our embedding payload contains Markdown alone. Attribution stays in the observation layer so a changed title or newly discovered author doesn't force another embedding of unchanged text.

HTML, alternate text representations, collection timestamps, and URL variants stay out of that payload. Keeping every representation in the prompt would spend tokens repeating the page in different clothes.

Truncation is also off here. Otherwise, distinct pages with identical retained prefixes could share a content key even though their full bodies differ.

This Example input uses placeholder documentation addresses; replace them with the pages you collect.

import hashlib
import json
import os
from urllib.parse import urlsplit
from uuid import uuid4

from apify_client import ApifyClient

run_input = {
    "query": "",
    "startUrls": [
        "https://example.com/docs/",
        "https://example.com/docs/?edition=stable"
    ],
    "outputFormats": ["markdown"],
    "includeMetadata": True,
    "chunkMarkdown": False,
    "maxContentLength": 0,
    "maxCrawlDepth": 0
}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("vexaloom-labs/rag-web-browser").call(
    run_input=run_input
)
if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Collection stopped: {run['status']}")

items = client.dataset(run["defaultDatasetId"]).iterate_items()

Keep query parameters intact. An edition, locale, or document identifier can live there; removing the query string is an application decision with possible data loss.

This source key deliberately treats different address strings as different sources. Any later alias merging needs evidence beyond their appearance.

Hash the bytes you actually embed

The normalization policy is deliberately boring: no stripping, whitespace collapse, Unicode normalization, or line-ending conversion.

The code encodes the returned Markdown as UTF-8 and hashes those exact bytes. Its whitespace check only rejects empty content; it doesn't modify the payload.

The additional keys below belong to our application records. Every observation gets its own identifier, while its source key remains the collected url.

This cell writes unique content to content.jsonl and every collection observation to source_observations.jsonl.

observation_fields = (
    "url", "loadedUrl", "redirectedUrl", "canonicalUrl",
    "contentHash", "scrapedAt", "title", "author",
    "publishedDate", "httpStatusCode", "fetchError"
)

contents = {}
observations_by_content = {}

with (
    open("content.jsonl", "w", encoding="utf-8") as content_file,
    open("source_observations.jsonl", "w", encoding="utf-8") as source_file,
):
    for item in items:
        source_url = item.get("url")
        body = item.get("markdown")
        status = item.get("httpStatusCode")

        observation = {
            "observation_id": uuid4().hex,
            "source_key": source_url,
            "content_key": None,
            **{
                field: item[field]
                for field in observation_fields
                if field in item
            },
        }

        indexable = (
            isinstance(source_url, str)
            and bool(source_url)
            and isinstance(body, str)
            and bool(body.strip())
            and isinstance(status, int)
            and 200 <= status < 300
            and not item.get("fetchError")
        )

        if indexable:
            content_key = hashlib.sha256(
                body.encode("utf-8")
            ).hexdigest()

            if content_key in contents:
                if contents[content_key] != body:
                    raise RuntimeError("Content-key collision")
            else:
                contents[content_key] = body
                content_file.write(json.dumps({
                    "content_key": content_key,
                    "embedding_text": body,
                }, ensure_ascii=False) + "\n")

            observation["content_key"] = content_key
            observations_by_content.setdefault(
                content_key, []
            ).append(observation)

        source_file.write(
            json.dumps(observation, ensure_ascii=False) + "\n"
        )

The embedding worker reads embedding_text and uses content_key as its deduplication key. It must preserve that payload contract. If you later prepend headings or normalize whitespace, make that transformation before hashing.

The observation file also retains records excluded from indexing. A failed fetch shouldn't become evidence for an answer, but it still explains a gap in your corpus.

These files represent a fresh export: opening them replaces their previous contents. The lookup dictionaries also live in memory, so this example doesn't provide deduplication across separate executions.

For a persistent corpus, move uniqueness enforcement and source relationships into database tables, and commit content plus observations together. A partially written relationship is another quiet citation failure.

Serialize citations after retrieval

Embedding input and answer context have different needs.

Embedding the same body with every source URL attached would defeat this deduplication policy. After retrieval, however, the model needs approved citation addresses and available attribution alongside the retrieved body.

Our citation policy prefers the loaded address, then the redirected address, then the collected address. It accepts HTTPS addresses on explicitly approved hosts. canonicalUrl remains available for inspection without automatically receiving citation authority.

This serializer returns a JSON context block for a retrieved content key, or nothing when no associated observation has an approved address.

def prompt_context(
    content_key: str,
    approved_hosts: set[str],
) -> str | None:
    allowed_hosts = {host.lower() for host in approved_hosts}
    citations = []

    for observation in observations_by_content[content_key]:
        citation_url = None

        for field in ("loadedUrl", "redirectedUrl", "url"):
            candidate = observation.get(field)
            if not isinstance(candidate, str):
                continue

            try:
                address = urlsplit(candidate)
                approved = (
                    address.scheme == "https"
                    and address.hostname in allowed_hosts
                    and address.username is None
                    and address.password is None
                )
            except ValueError:
                continue

            if approved:
                citation_url = candidate
                break

        if citation_url is not None:
            citations.append({
                "observation_id": observation["observation_id"],
                "url": citation_url,
                **{
                    field: observation[field]
                    for field in ("title", "author", "publishedDate")
                    if field in observation
                },
            })

    if not citations:
        return None

    return json.dumps({
        "citations": citations,
        "content": contents[content_key],
    }, ensure_ascii=False)

The observation identifiers let your answer layer resolve citations back to collection records. The body appears once in each context block, while its source relationships remain explicit.

Repeated collection observations can still make citation metadata bulky. You can collapse equivalent citations for display while retaining their observation identifiers in storage. Presentation deduplication doesn't need to rewrite provenance.

Treat the serialized page content as untrusted evidence in your prompt. JSON provides structure; it doesn't make instructions embedded in a page trustworthy.

What this split doesn't solve

Exact hashing doesn't recognize equivalent prose with different formatting. A changed line ending produces a different key under this policy. That's conservative, and sometimes annoyingly expensive.

Whole-page deduplication also misses shared sections inside otherwise different pages. If you move to chunk embeddings, apply the same rule to the exact chunk payload and retain its parent observation relationship. Heading context included in the embedding belongs inside the hash boundary too.

Nor does identical text establish interchangeable authority. A copied page and its original source can share an embedding payload while carrying different publication context and citation value. The source layer is where you retain that distinction.

The Actor keeps pages that return content with an error status. Our indexing policy excludes them, because an error page can otherwise become a confidently retrieved answer about the wrong thing.

Finally, timestamps alone don't establish the latest authoritative version. If collection order matters, preserve the run identifier or ingestion batch in your storage system rather than guessing from a missing date.

The schema supports separate identities

Search & URL Markdown Extractor for LLMs exposes enough address and attribution information to preserve source observations independently of embedding content.

The verdict: the schema supports this separation, but it doesn't establish a universal document key. Keep the Actor's change signal, define your own embedding identity, and retain the source-to-content relationships needed for auditing.

Updates then become explicit relationships. A source can point to changed content without overwriting its earlier observation. Unchanged content can reuse an embedding without erasing another source.

Deleting an old embedding becomes a reference check: does any retained observation still need it? A changed page alone isn't permission to remove shared content.

Trace a retrieved body back to its sources

Replace the placeholder addresses with your collection targets.
Set APIFY_TOKEN in the environment and execute the Python cells in order.
Check that repeated embedding payloads share a content record while their observations remain separate.
After retrieval, call prompt_context with a content key and your approved host set.

Use this exact Example input to collect only the supplied pages:

{
  "query": "",
  "startUrls": [
    "https://example.com/docs/",
    "https://example.com/docs/?edition=stable"
  ],
  "outputFormats": ["markdown"],
  "includeMetadata": true,
  "chunkMarkdown": false,
  "maxContentLength": 0,
  "maxCrawlDepth": 0
}

A smaller vector index doesn't get editorial control over your source ledger.