合併重疊的爬取結果以避免重複計算：1,869 行輸入、1,671 行輸出

Most of my scraping jobs don't produce one clean dataset. They produce several that overlap: the same search run with four keywords, the same category page scraped every night, two scrapers pointed at sites that list the same products. Stack them up and count, and you count some things twice.

The fix is boring, which is probably why it gets skipped: merge everything, pick a key, keep one row per key. This post walks through that on a real example with real numbers, first in plain Python so you can see the logic, then with an Apify Actor I wrote for the case where the data already lives in Apify datasets.

Disclosure: I built the Actor used below (Dataset Deduplication & Merge) and I get paid when other people run it. This article was written with AI assistance. The numbers come from one real run on 2026-11-05 (run id below), and I ran the code against the live APIs before posting.

The example: four keyword searches on Hacker News

I wanted every Hacker News story from the last 120 days about databases. The Algolia HN API is public and needs no key, but it's keyword search, so I ran four queries: sqlite, postgres, duckdb and database.

A story titled "Databases in Depth: SQLite, DuckDB, PostgreSQL..." matches all four. If I just concatenate the results, it shows up four times, and any "top stories by points" chart I build later is wrong.

Here's the fetch. Each query's hits go into their own Apify dataset, which is roughly what you'd have after four scraper runs:

import time, requests

TOKEN = "YOUR_APIFY_TOKEN"
API = "https://api.apify.com/v2"
H = {"Authorization": f"Bearer {TOKEN}"}

now = int(time.time())
since = now - 120 * 86400  # last 120 days
fields = ["objectID", "title", "url", "author", "points", "num_comments", "created_at"]

dataset_ids = []
for query in ["sqlite", "postgres", "duckdb", "database"]:
    hits = requests.get("https://hn.algolia.com/api/v1/search", params={
        "query": query,
        "tags": "story",
        "hitsPerPage": 1000,
        "numericFilters": f"created_at_i>{since},created_at_i<{now}",
    }, timeout=60).json()["hits"]
    rows = [{**{f: h.get(f) for f in fields}, "query": query} for h in hits]

    ds = requests.post(f"{API}/datasets", headers=H, timeout=60).json()["data"]
    requests.post(f"{API}/datasets/{ds['id']}/items", headers=H, json=rows, timeout=60).raise_for_status()
    dataset_ids.append(ds["id"])
    print(query, len(rows), ds["id"])

On 2026-11-05 that gave me 292 rows for sqlite, 646 for postgres, 165 for duckdb and 766 for database: 1,869 rows in total. The search index changes daily, so your counts will be a little different.

I added a query field to every row on purpose. After deduplication it tells you which search found the story first, which turns out to be handy.

Picking the key

This is the decision that matters, and it depends on what "the same thing" means for your data.

• objectID is HN's own story id. Two rows with the same id are the same story, full stop. That's the safe key.
• url is tempting, because the same blog post often gets submitted more than once under different ids. But Ask HN and Show HN text posts have no URL at all, and you don't want every URL-less story collapsed into one row.

I started with objectID. I'll come back to url at the end.

If all the rows fit in one list in memory (call it all_rows), this would be the whole job:

seen, unique = set(), []
for row in all_rows:
    if row["objectID"] not in seen:
        seen.add(row["objectID"])
        unique.append(row)

It stops being that simple when the rows are spread across several datasets, there are a few million of them, you want to know how many times each key appeared, or the key is two fields together (shop + SKU, say). That's the part the Actor handles. It reads the datasets in order, keeps the key index on disk in SQLite instead of in memory, and writes one row per key with a count.

Running the merge

Input:

{
  "datasetIds": ["<sqlite>", "<postgres>", "<duckdb>", "<database>"],
  "keyFields": ["objectID"],
  "keep": "first",
  "trim": true,
  "ignoreEmpty": true,
  "demoMode": false
}

keep: "first" keeps the earliest occurrence in the order you list the datasets; "last" keeps the latest, which is what you want when later datasets are fresher snapshots. trim strips whitespace from key values before comparing. ignoreEmpty keeps rows with a missing or empty key as separate rows instead of merging them all into one. keyFields takes several fields if you need a composite key, and dotted paths like product.sku for nested values.

You can paste that into the Actor page and press Start. From code, continuing the script above:

run_input = {
    "datasetIds": dataset_ids,
    "keyFields": ["objectID"],
    "keep": "first",
    "trim": True,
    "ignoreEmpty": True,
    "demoMode": False,
}
run = requests.post(
    f"{API}/acts/ahmed.fouaad82~dataset-dedup-merge/runs",
    headers=H,
    json=run_input,
    params={"waitForFinish": 60, "maxTotalChargeUsd": 0.10},
    timeout=90,
).json()["data"]
while run["status"] in ("READY", "RUNNING"):
    time.sleep(5)
    run = requests.get(f"{API}/actor-runs/{run['id']}", headers=H, timeout=60).json()["data"]
print(run["status"], run["id"])

summary = requests.get(
    f"{API}/key-value-stores/{run['defaultKeyValueStoreId']}/records/SUMMARY", headers=H, timeout=60
).json()
print(summary)

waitForFinish tops out at 60 seconds on the Apify API, hence the polling loop. maxTotalChargeUsd is a hard spending cap for the run.

My run (id GGii3VrqS3GQZG1FW) took 5.4 seconds and printed this summary:

{
  "processedItems": 1869,
  "outputItems": 1671,
  "uniqueItems": 1671,
  "duplicateItems": 198,
  "sourceDatasets": 4,
  "stopReason": "completed",
  "complete": true
}

So 198 of the 1,869 rows, about 11%, were the same story found by more than one query. Not dramatic, but enough to skew any count or ranking.

Reading the output

Each output row wraps the original record and adds three fields:

{
  "item": {
    "objectID": "49010949",
    "title": "Show HN: SyncLite – An embedded synchronization layer for SQLite and DuckDB",
    "url": "https://github.com/syncliteio/SyncLite",
    "author": "mahcha",
    "points": 2,
    "num_comments": 4,
    "created_at": "2026-07-22T18:06:47Z",
    "query": "sqlite"
  },
  "occurrences": 4,
  "sourceDatasetId": "oUicy9wRLdWsHim8l",
  "sourceIndex": 103
}

occurrences is the number of source rows that shared the key. sourceDatasetId and sourceIndex say exactly where the kept row came from, so you can always trace a row back to its run.

occurrences doubles as a free signal. Here it's "how many of my queries matched this story", a rough relevance score:

items = requests.get(
    f"{API}/datasets/{run['defaultDatasetId']}/items",
    headers=H, params={"limit": 5000}, timeout=60,
).json()

repeats = sorted((i for i in items if i["occurrences"] > 1), key=lambda i: -i["occurrences"])
print(len(items), "rows,", len(repeats), "stories matched more than one query")
for i in repeats[:3]:
    print(i["occurrences"], i["item"]["title"])

1671 rows, 174 stories matched more than one query
4 Show HN: SyncLite – An embedded synchronization layer for SQLite and DuckDB
4 Databases in Depth – SQLite · DuckDB · PostgreSQL · MariaDB · ClickHouse · Mongo
4 Show HN: Spice 2.0 – Real-Time Analytical Query on Operational Data, Without ETL

The full spread was 1,497 stories seen once, 154 seen twice, 16 three times and 4 by all four queries. If you only want to audit what got merged, set "onlyDuplicates": true and you get just the repeated keys (174 rows here) instead of the full cleaned set.

To get a flat CSV instead of nested JSON, export the output dataset with unwind=item, or flatten item yourself.

Second pass: the same link submitted twice

Deduplicating by id removed the query overlap, but HN has another kind of duplicate: one article submitted several times by different people, each with its own id. A quick local count on the 1,671 rows:

from collections import Counter
urls = Counter((i["item"].get("url") or "").strip().lower() for i in items)
urls.pop("", None)
print(sum(urls.values()) - len(urls), "rows share a URL with an earlier row")

That prints 222. One ClickHouse blog post about Postgres appeared 11 times. Whether those count as duplicates depends on the question. For "which articles did people share", yes. For "which submissions got traction", no, because each submission has its own points and comments.

If you want them merged, you can feed the first run's output dataset straight back in. The key is now nested under item, so it's a dotted path:

{
  "datasetIds": ["<output dataset of the first run>"],
  "keyFields": ["item.url"],
  "caseInsensitive": true,
  "trim": true,
  "ignoreEmpty": true,
  "keep": "first"
}

I didn't run this second pass for the post, so the 222 comes from the Python count above, not from the Actor. Two things to know before you try it. With ignoreEmpty on, the 114 stories without a URL stay as separate rows instead of collapsing into one. And the Actor only trims and lowercases. It doesn't normalise URLs, so http:// vs https://, a trailing slash or a utm_ parameter still count as different keys. Lowercasing can also, in theory, merge two URLs whose paths differ only in case. Clean the URLs into a separate field first if that matters to you.

What it cost

The Actor charges $0.02 per 1,000 source rows read, and the first 50 rows of each run are free. A run like mine would cost (1,869 − 50) × $0.00002 = $0.036. It doesn't use a browser or proxies, so it finishes in a few seconds. The Pricing tab on the Actor page is the authoritative number.

Limits

• It only reads Apify datasets you pass in, and it never modifies them; output goes to the run's own dataset. If your data is in a CSV, push it into a dataset first (the POST /datasets/{id}/items call above does that).
• It matches exact key values after optional trimming and lowercasing. There's no fuzzy matching, so "Acme Inc." and "ACME, Inc" stay different.
• Run IDs: with limited permissions, a run ID doesn't grant access to a private run's dataset. Pass dataset IDs instead. The Actor README shows how to turn run IDs into dataset IDs with your own token first.
• The disk index lives only for the run. If a run is aborted, start a fresh one rather than resuming.
• I've tested it at 100,000 rows. Millions of rows are what it's designed for, but I haven't measured that yet.

If you run it on your own data and something merges that shouldn't (or doesn't merge when it should), open an issue on the Actor page with the input you used and I'll look at it.