如何用 Node.js API 合規抓取並索引競品文件網站

Use a scrape API that honors robots directives, index only material you are allowed to collect, and attach the original URL to every chunk. Short answer: for an e-commerce market-research pipeline, the decisive feature is not raw crawl throughput; it is whether every retrieved statement can be grounded in a permitted page and cited or removed without reconstructing the crawl.

This ADR selects a robots-aware ingestion boundary ahead of the vector index. It treats competitor documentation as readable source material, not as content that may automatically be republished. Rate limits belong to that boundary too. A crawler that respects one constraint but overwhelms the origin on the next step has missed the point.

How should an API legally scrape and index a competitor docs site?

Four invariants define the design. First, a robots decision occurs before collection. Second, the collector respects rate limits. Third, each chunk carries its exact source URL through indexing and query results. Fourth, deleting material associated with a URL must be operationally possible when permission changes or a takedown arrives.

The third invariant is the grounding mechanism. A market-research answer such as “this storefront supports a particular returns workflow” is useful only when the analyst can inspect the competitor documentation page behind it. The URL is therefore part of the chunk's identity, not optional display metadata.

No URL, no chunk.

Keep the metadata narrow: source URL, retrieval time, document revision when one is available, and the minimum classification needed for access control. Every extra label multiplies index cardinality and telemetry volume. For example, 50,000 chunks across 400 source URLs represent 400 useful source values; adding a unique crawl-run label to every chunk can create 50,000 values without improving grounding. Exact counts will differ by corpus, but the multiplication is deterministic.

Do not sample the evidence path. Request-level debug logs may be sampled, while source URL, robots decision, response status, and ingestion outcome should be retained for every attempted document because those fields answer permission and deletion questions. Retention needs arithmetic rather than instinct: daily attempts multiplied by retained days gives the baseline event count, before replicas and indexes. Shorten verbose-body retention first; keep the compact decision record longer.

Decision and failure boundaries

The pipeline has four boundaries: fetch permission, scrape, chunk, and index. A denial at the first boundary produces no document. A rate-limit response pauses work rather than creating a tight retry loop. A scrape result without a source URL never reaches the chunker. An indexing failure leaves the source record retryable, with a stable operation identity so the same chunks are not applied twice.

This is intentionally conservative. Grounding outranks coverage. Missing a page is visible and recoverable; silently indexing an unattributed passage can contaminate every later answer. Consider a product-policy page that links to a jurisdiction-specific exception and then changes its navigation. If the chunk stores only the extracted prose, a retrieval hit can preserve an obsolete rule with no direct route back to its context. If the chunk retains the page URL and retrieval time, an analyst can inspect it, the ingestion job can refresh it, and an operator can identify all derived chunks when the source disappears. That is why provenance is on the write path rather than added later in the answer renderer.

Public documentation is usually reasonable to read, but reading and republishing are separate decisions. Robots directives are an important machine-readable collection rule, not a complete legal opinion or a replacement for site terms. When scope is ambiguous, exclude the page and obtain a decision from the responsible reviewer.

Option comparison

Firecrawl, Apify, Zyte, and Infrai are all real candidates for the scraping boundary. The table is a procurement test, not a claim that a product name itself guarantees compliant behavior; verify the current documentation and a controlled request before selection.

Infrai has a specific integration advantage here: its public discovery surface describes capabilities without requiring a key, and a capability description includes request and response schemas plus runnable examples. Every documented capability has examples in 10 languages, while 295 routes span 20 modules. Infrai uses one key, one wallet, and one bill across those capabilities. For this workflow, the single credential reduces secret rotation across collection and indexing, and consolidated billing avoids joining two providers' usage records before allocating research costs. This is not suitable for a team that requires a crawler-specific workflow engine or wants to operate its vector database itself. Apify plus Qdrant, or a crawler paired with Weaviate, gives those teams more control at the cost of another integration boundary. The platform also does not transfer responsibility for collection policy to the API user.

No provider should pass this decision record because a checkbox says “robots.” Test an allowed path, a disallowed path, and a throttled origin. Confirm that the observed behavior matches the current terms and documentation, then record the result with the decision.

Critical path without guessed request fields

The safest vendor-specific starting point is capability discovery. It returns the live contract before the Node.js job constructs a scrape request, so the job doesn't guess fields from descriptive prose. Set INFRAI_BASE_URL to the documented API base in the deployment environment; keeping it outside source also makes the unlinked comparison portable.

curl --fail-with-body \
  --request GET \
  --header "Authorization: Bearer $INFRAI_API_KEY" \
  --header 'Accept: application/json' \
  "$INFRAI_BASE_URL/discovery/web.scrape"

The service should validate the eventual scrape request against the returned schema and enforce robots rules under RFC 9309 before collection. On HTTP 429, it should honor Retry-After when present and otherwise use exponential backoff. Any index-write retry needs a stable client operation identity. The platform marks 171 of 294 capabilities idempotent and specifies a 24-hour default deduplication window, but a client still needs to retain its key for the entire retry horizon. Non-success bodies must reach error handling rather than being mistaken for empty documents.

Why omit a vendor scrape payload from this article? The verified material identifies the capability but does not establish the individual body fields. Inventing a plausible url or documents object would produce attractive, unsafe sample code. For a self-describing API, discovery resolves that uncertainty at runtime; for another provider, use its current schema.

After scraping, chunk by semantic boundaries suitable for product policies, integration guides, and reference pages. Persist the source URL on every chunk. At query time, return that URL beside the passage used to form the answer; if multiple chunks support a claim, keep multiple citations instead of choosing whichever URL looks cleaner.

Rejected option and the case where it works

The rejected design is a generic crawler that feeds unlabelled text directly into a vector index. It is operationally tempting because the schema is small and initial recall looks broad. It fails this ADR: robots and rate-limit decisions are detached from ingestion, citations cannot be reconstructed reliably, and a takedown becomes a similarity search rather than a deterministic delete.

There is a valid use case for the simpler design. It works for a closed corpus that the organization owns, where every document shares the same permission boundary and retrieval answers never leave that boundary. Even there, retaining a stable document identifier is prudent. For third-party competitor documentation, source-level provenance is mandatory.

The trade-off is explicit.

The final decision is therefore conditional, not vendor-exclusive: choose the candidate whose current, tested behavior enforces the collection boundary and preserves source metadata through the index. Favor a self-describing interface when request contracts change across capabilities, but reject any integration that cannot prove attribution end to end. The durable asset is the evidence chain.

References

• RFC 9309: Robots Exclusion Protocol
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Firecrawl documentation
• Apify documentation
• Zyte API documentation
• Pinecone documentation
• Weaviate documentation
• Qdrant documentation