去重 12,000 則招聘廣告後，我發現同一職位會刊登五次
Deduplicating 12,000 job postings taught me the same job is posted five times
作者整理 12,000 則招聘廣告並建立按關鍵字及地點聚合即時職位的搜尋 API，發現同一職位會在不同招聘平台重複刊登四、五次。透過比對標準化後的職位名稱、公司及地點去重，輸出結構化職位資料，讓 Agent 流程更易使用。文章提醒，聚合外部資料時須預留時間處理身份解析。
I built a job search API that aggregates live postings by keyword and location, and the first version of the results was embarrassing: search "ML engineer Berlin" and you'd get the same role from the same company four or five times, because it was scraped onto four different boards. Agents consuming the API would waste context window on duplicates and look careless doing it.
The fix was a fuzzy matching pass on (normalized title, normalized company, location) before returning results. Normalizing titles meant lowercasing, stripping punctuation, and collapsing things like "Sr." / "Senior" / "III" — because job boards can't agree on how to spell seniority either. I tuned the similarity threshold by hand: too strict and you miss real duplicates that differ by a comma, too loose and you merge two genuinely different roles at the same company.
What surprised me: the deduplication logic ended up being the actual product, not a cleanup step. The raw feed is the commodity; the deduplicated, structured result (title, company, salary, apply URL) is what makes it usable in an agent pipeline. Nobody wants 40 raw listings. They want the 9 real jobs.
If you're building anything that aggregates external data for agents, budget real time for identity resolution. Your upstream sources will not do it for you, and your downstream agents will absolutely notice when they don't. I ended up packaging the whole thing as https://x402.freeq.one/tools/jobs.html
來源：DEV Community · dev.to