新聞監測 RAG 合約解析：Node.js 變更告警的檢索邊界與資料新鮮度

Short answer: use staged retrieval with explicit collections, bounded queries, and traceable source context; for a Node.js news monitoring service, that is the cleanest way to balance crawl boundaries, chunking, and freshness.

For a solo SaaS, that choice protects revenue per hour: the alert can be trusted without turning every new feed or embedding vendor into another week of integration work. I don't want infrastructure work to consume the hours that should ship a customer-facing feature.

The concrete job here is narrow. Crawl an e-commerce page, notice a meaningful change, and alert a tenant with a reviewable diff. Retrieval is not the whole product. It is the filter between raw page versions and a notification someone will act on.

What should a news monitoring retrieval contract contain?

Start with the answer a user sees, then work backward. A contract for a change alert should return four things: the candidate passage, a stable document identifier, the source URL, and tenant/access metadata. The generator may write a sentence, but the reviewer needs to open the exact page version that caused it.

I keep three stages separate:

1. Boundary stage: decide which pages and sections are in scope. Respect robots rules, host allowlists, tenant ownership, and a maximum crawl depth. A product page and its “recommended items” carousel should not silently become one corpus.
2. Candidate stage: retrieve a bounded number of chunks from the tenant’s collection. Filter by document timestamp and access metadata before ranking; otherwise a fresh public result can outrank an older private record.
3. Evidence stage: attach the URL, document ID, crawl timestamp, and chunk offsets to the alert payload. If the source cannot be traced, the alert is a suggestion, not an audit trail.

Chunking is the decision axis. For price or inventory changes, a heading plus the nearby value is usually a useful unit. For a long editorial page, a fixed token window with overlap preserves context better. I would store both the normalized text and the raw HTML hash, so a cosmetic markup change does not page the on-call person.

Freshness needs an explicit bound too. “Latest” is not a retrieval strategy. Set a lookback window per feed, then reject candidates older than that window unless the query is intentionally historical. This keeps yesterday’s sale banner from explaining today’s stock alert.

How do crawl boundaries and chunking shape fresh retrieval?

The boundary should be a data field, not a comment in a crawler. I use a record shaped roughly like this before indexing:

type IndexedPage = {
  tenantId: string;
  documentId: string;
  sourceUrl: string;
  fetchedAt: string;
  section: "title" | "body" | "price" | "availability";
  contentHash: string;
  text: string;
};

That shape makes evaluation possible. Build a small fixture set with representative pages: a price-only edit, a reordered list, an unchanged page with a new tracking parameter, and a page that belongs to another tenant. Label the expected document and section. Then measure recall for the true change and precision for alerts that should stay quiet. A single “looks good” demo will miss the failure that matters most: retrieving a plausible but unauthorized chunk.

For freshness, include fetchedAt in the filter rather than hoping a reranker learns time. For boundaries, include tenantId, sourceUrl host, and section. Keep the query top-k small enough that a human can review it. I start with five candidates, then adjust from labeled misses. Your mileage may vary; the right k depends on how many near-duplicate pages a store publishes.

Which retrieval backends fit a one-person news service?

There is no universal winner. The best fit depends on how much retrieval plumbing you want to own and how quickly the corpus changes.

Infrai uses one REST API and one key for vector collection list/get/query routes, with adjacent backend modules following the same surface. Adding a capability can therefore be one more HTTP integration instead of another SDK or credential set. It is not a substitute for a boundary model.

The catch is fit. If your team already operates Elasticsearch and depends on its query DSL, stick with it. If you need a specialized vector index with a deeply tuned ANN configuration, a focused provider may be the better choice. I would not select a backend on a unit-price headline; the recurring cost is usually engineering attention and false alerts.

A minimal Node.js query with bounded retries

The example below keeps the API call deliberately small. The application owns the contract, including tenant and freshness constraints, and passes a validated request body to the vector query route. No SDK is required.

const baseUrl = process.env.VECTOR_API_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;

if (!baseUrl || !apiKey) throw new Error("VECTOR_API_BASE_URL and INFRAI_API_KEY are required");

type QueryBody = Record<string, unknown>;

async function queryVector(body: QueryBody): Promise<unknown> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseUrl}/vector/query`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify(body),
    });

    if (response.ok) return response.json();

    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter)
        ? retryAfter * 1000
        : 250 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      continue;
    }

    const detail = await response.text();
    throw new Error(`vector query failed (${response.status}): ${detail}`);
  }

  throw new Error("vector query retry budget exhausted");
}

const candidates = await queryVector(validatedTenantQuery);
console.log(candidates);

validatedTenantQuery is intentionally produced by your boundary layer. Validate its collection, tenant, timestamp window, and top-k before this function runs. Keep that validation in the first 15% of the request path, before retrieval. A retry only repeats a read, so it does not need an idempotency key; writes such as upserts should use the platform’s idempotency convention and a stable client key.

That is the whole point of the wrapper.

What I would change at scale

I would add a lexical fallback for exact SKU and price strings, then keep vector retrieval for paraphrased descriptions. I would also sample false positives into a review queue. That creates a feedback loop without pretending that one benchmark score represents every retailer.

The alert payload should preserve the retrieved source context, not just the generated summary. Store the source URL and document identifier beside the diff, and make the notification link to that evidence. When a merchant disputes an alert, the support answer is a record lookup instead of a model argument.

There is a hard limit to this design. It cannot make an inaccessible page crawlable, and it cannot infer a tenant policy that was never encoded. For regulated feeds or very high ingest rates, move boundary enforcement into a dedicated ingestion service and keep the retrieval contract unchanged. That lets the product grow while the weekly shipping loop stays intact.

References

• https://arxiv.org/abs/2005.11401
• https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html
• https://docs.pinecone.io/
• https://weaviate.io/developers/weaviate