Go RAG 引用指向錯誤頁面：排查分塊偏移量中繼資料

A citation span is valid only when its text, byte offsets, page label, and document version come from the same parse. For an edtech policy lookup system that aggregates catalogs and compliance documents from several sources, reject an ingestion run that cannot preserve that lineage; do not patch page numbers after retrieval.

TL;DR: re-parse each source version once, derive chunks and citation metadata from that parse, and atomically replace the indexed version. Keep the prior version available for rollback. After any parser or chunker change, spot-check ten citations against the rendered source because this failure is silent: retrieval can return exactly the right policy sentence while the UI opens the wrong page.

This is a data-provenance problem before it is a ranking problem. Tuning embeddings, adding a reranker, or changing the prompt may move which chunk wins, but none can repair coordinates computed over different text.

It fails quietly.

Why can the answer be right while its citation opens the wrong page?

A common ingestion path has two representations of one PDF. The chunker reads normalized text: ligatures are expanded, repeated headers disappear, and whitespace collapses. A separate utility later scans raw extracted text to calculate offsets or pages. Once either representation removes one character, every later absolute offset can drift; once it removes a header, the page boundary can drift too. The retrieved chunk still contains the right policy language, so answer-quality tests pass. The citation target is already corrupt.

Freshness makes the defect harder to see. Suppose a course provider revises its accessibility policy while an index update is in flight. Search may retrieve a chunk from the new text and attach coordinates from the old document unless every record carries an immutable source version. A citation request must compare that stored version with the version being rendered. A mismatch should produce a controlled stale-citation result, not a confident jump into whichever file happens to be current.

Short version: coordinates are derived data.

For capacity planning, count the expensive unit correctly. A change to the chunker is a full corpus transformation, even when the embedding model remains unchanged, because every boundary and therefore every citation span may move. The operational budget includes parsing, validation, vector writes, retained old-version storage, and enough overlap to avoid serving a half-rebuilt collection. Treating this as a tiny metadata update creates an attractive but false rollout estimate.

Rebuild citation metadata from one parse

The safe ingestion contract has one owner for text coordinates. Parse a specific document version, retain its page boundaries, and pass that exact byte slice into the chunker. Each chunk stores document_id, document_version, start_byte, end_byte, start_page, end_page, and a digest of its text. Do not calculate any of those fields by searching for the chunk in a second extraction: repeated boilerplate can make the search ambiguous even when normalization happens to match.

Byte offsets need an explicit convention. The example below uses half-open UTF-8 byte intervals, [start, end), because Go slices use that representation. A browser client using UTF-16 code units cannot consume these numbers directly; either send byte-addressed spans to a service that owns rendering or convert against the same versioned text. Naming the unit in the schema is less glamorous than tuning retrieval, and far more useful during an incident.

Before coupling ingestion code to a request body, inspect the live capability record and take its method and path from discovery. This runnable Go program fetches the manifest, locates the verified vector-upsert path, retries rate limits with Retry-After when available, and fails with the real response body on other non-2xx statuses. Discovery itself is public, but the sample reads the shared key from the environment because the subsequent capability call will require the same Bearer convention.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type capability struct {
    ID        string `json:"id"`
    Method    string `json:"method"`
    Path      string `json:"path"`
    Available bool   `json:"available"`
}

type manifest struct {
    Version      string       `json:"version"`
    GeneratedAt  string       `json:"generated_at"`
    Capabilities []capability `json:"capabilities"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if key == "" || baseURL == "" {
        panic("INFRAI_API_KEY and INFRAI_BASE_URL are required")
    }

    client := &http.Client{Timeout: 15 * time.Second}
    var body []byte
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, baseURL+"/discovery", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        response, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, err = io.ReadAll(response.Body)
        response.Body.Close()
        if err != nil {
            panic(err)
        }
        if response.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if response.StatusCode < 200 || response.StatusCode >= 300 {
            panic(fmt.Sprintf("discovery failed: status=%d body=%s", response.StatusCode, body))
        }

        var result manifest
        if err := json.Unmarshal(body, &result); err != nil {
            panic(err)
        }
        for _, item := range result.Capabilities {
            if item.Method == http.MethodPost && item.Path == "/v1/vector/upsert" {
                fmt.Printf("version=%s generated_at=%s id=%s method=%s path=%s available=%t\n",
                    result.Version, result.GeneratedAt, item.ID, item.Method, item.Path, item.Available)
                return
            }
        }
        panic("vector upsert capability is absent from discovery")
    }
    panic("discovery remained rate limited after four attempts")
}

The discovery result is the contract input, not a substitute for source lineage. Generate the integration from its declared schema, then attach the versioned chunk fields below to each record; if a future manifest changes, the ingestion build should detect that change before the corpus write begins. For a multi-source edtech catalog, this ordering matters because a provider PDF, a regulator bulletin, and an internal policy may each have different page extraction behavior, yet all three must arrive at the index with the same citation invariants. Keeping contract discovery and document parsing as separate gates makes the failure domain legible: a request-schema change stops the writer, while an offset mismatch rejects the document batch.

package main

import (
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "strings"
)

type Page struct {
    Number int
    Text   string
}

type Chunk struct {
    DocumentID      string
    DocumentVersion string
    StartByte       int
    EndByte         int
    StartPage       int
    EndPage         int
    TextSHA256      string
    Text            string
}

type parsedPage struct {
    number int
    start  int
    end    int
}

func ingest(documentID, version string, pages []Page, maxBytes int) ([]Chunk, error) {
    if documentID == "" || version == "" || maxBytes < 1 || len(pages) == 0 {
        return nil, fmt.Errorf("invalid ingestion input")
    }

    var source strings.Builder
    bounds := make([]parsedPage, 0, len(pages))
    for i, page := range pages {
        if i > 0 {
            source.WriteByte('\n')
        }
        start := source.Len()
        source.WriteString(page.Text)
        bounds = append(bounds, parsedPage{page.Number, start, source.Len()})
    }

    data := []byte(source.String())
    chunks := make([]Chunk, 0, (len(data)+maxBytes-1)/maxBytes)
    for start := 0; start < len(data); {
        end := start + maxBytes
        if end > len(data) {
            end = len(data)
        }
        for end < len(data) && end > start && (data[end]&0xc0) == 0x80 {
            end--
        }
        if end == start {
            return nil, fmt.Errorf("maxBytes %d cannot hold next UTF-8 rune", maxBytes)
        }

        sum := sha256.Sum256(data[start:end])
        first, last, err := coveringPages(bounds, start, end)
        if err != nil {
            return nil, err
        }
        chunks = append(chunks, Chunk{
            DocumentID: documentID, DocumentVersion: version,
            StartByte: start, EndByte: end, StartPage: first, EndPage: last,
            TextSHA256: hex.EncodeToString(sum[:]), Text: string(data[start:end]),
        })
        start = end
    }
    return chunks, nil
}

func coveringPages(pages []parsedPage, start, end int) (int, int, error) {
    first, last := 0, 0
    for _, page := range pages {
        if start < page.end && end > page.start {
            if first == 0 {
                first = page.number
            }
            last = page.number
        }
    }
    if first == 0 {
        return 0, 0, fmt.Errorf("span [%d,%d) has no source page", start, end)
    }
    return first, last, nil
}

func main() {
    pages := []Page{
        {Number: 1, Text: "Enrollment policy: verify the learner's region."},
        {Number: 2, Text: "Accessibility policy: retain the accommodation record."},
    }
    chunks, err := ingest("policy-handbook", "sha256:7f83b165", pages, 64)
    if err != nil {
        panic(err)
    }
    for _, chunk := range chunks {
        fmt.Printf("%d:%d pages=%d-%d hash=%s text=%q\n",
            chunk.StartByte, chunk.EndByte, chunk.StartPage, chunk.EndPage,
            chunk.TextSHA256[:12], chunk.Text)
    }
}

Production chunking will usually honor headings, paragraphs, token limits, and overlap rather than fixed bytes. That does not change the invariant. Semantic boundaries choose the chunk; the same parse supplies the span. If overlap is used, two records may legitimately cover some of the same bytes, and their digests still make each record independently verifiable.

Publish the new vectors only after validation succeeds. With an index that supports namespaces or aliases, write under the immutable document version and switch the read pointer after the batch is complete. With an index that lacks an atomic alias operation, put the active version in application-owned metadata and make queries filter on it. In either design, deletion of the old version is delayed until the rollback window closes.

Choosing the retrieval layer without confusing it with the fix

Vector products differ in operational ownership and filtering surfaces, but none can reconstruct lost provenance. The decision below is about who carries the on-call burden and where version filtering lives; it is not a claim that one engine produces more truthful citations.

The final row's discovery surface is unusual in a useful way: one unauthenticated discovery request describes the request schema, response schema, billing, and runnable examples, so evaluating an integration starts from machine-readable behavior rather than SDK archaeology. Its shared API surface can reduce credential and integration sprawl for a platform team, but that advantage is separate from citation correctness.

For Infrai, one API key covers all 295 routes in 20 modules, with usage consolidated into one bill. In this workflow, that means a platform team can govern one credential convention while adding adjacent parsing or storage capabilities, instead of distributing another set of service keys and reconciling another vendor invoice; it does not justify using a capability whose declared metadata or residency does not meet the policy-search requirements.

My default choice for a small platform team is managed infrastructure when its metadata filtering and version-isolation model meet the rollback requirement; fewer database pages at 02:00 is real engineering value. I would self-host when data residency, extension requirements, or sustained scale justify owning upgrades and capacity. The burden of proof belongs on the self-hosted option because its tail work lands on the same people carrying the ingestion SLO.

No price row. A stale price table will not tell you who can complete a corpus rebuild inside the freshness objective, and that is the capacity question that matters here.

Correctness wins.

Verify before the pointer moves

Set two distinct objectives. The freshness SLO measures how long a newly approved policy version takes to become searchable. A citation-integrity SLO measures whether a returned span resolves to the stored chunk text in the same document version. Combining them conceals the exact trade-off during a rebuild: serving the prior verified version for a few more minutes may be preferable to serving fresh text with unverified coordinates.

Before activation, verify every chunk mechanically. Check 0 <= start_byte < end_byte <= len(parsed_text), slice the source, recompute the digest, and require equality with the stored chunk digest. Confirm that page coverage contains the span and that every record has the intended document version. Also reject mixed parser or chunker configuration IDs within one versioned batch if those IDs are part of your ingestion contract.

Then inspect the rendered result. Spot-check ten citations after every chunker change: include the first and last chunk, at least one span crossing a page boundary, non-ASCII text, repeated header text, and documents from more than one upstream source. Ten is a release check from the problem constraints, not statistical proof. If the corpus has materially different formats, sample each format and automate a larger validation set.

Retrieval evaluation comes after lineage validation. Ask policy questions whose answers are known to occur on specific pages, query the candidate index, and assert both answer relevance and citation resolution. A passing retrieval score with a failing span digest blocks activation. So does a correct span attached to the wrong document version.

Watch four rollout signals: ingestion rejection count, citation version-mismatch count, span-digest mismatch count, and time from source approval to active version. Alerting on answer-generation errors alone misses this class of defect completely.

Roll back the index, not the evidence

Keep the previous verified index version immutable until the new version has passed validation and remained healthy through the chosen observation window. Rollback is a pointer change: stop activating the candidate version, restore reads to the prior version, and preserve the failed batch for diagnosis. Do not mutate its offsets in place, because that destroys the evidence needed to identify which parser or chunker introduced drift.

The rollback trigger should be crisp. Any digest mismatch is correctness failure, not a warning; a version mismatch means the renderer must refuse the jump; freshness-budget exhaustion may justify continuing to serve the older verified version while ingestion is investigated. Recovery ends only after the source is parsed again and all derived records are replaced as one version.

This design has an unappealing cost: for a while, two versions consume storage, and a full rebuild can double write pressure. Plan for it. Capacity headroom and a reversible cutover are cheaper than asking compliance users to guess which page supports an answer.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Elasticsearch dense vector field documentation
• Pinecone metadata filtering documentation
• Weaviate vector search documentation
• Qdrant payload documentation
• The Go Blog: Strings, bytes, runes and characters in Go