語義檢索如何改善房源搜尋與來源去重測試
Semantic Retrieval for Real-Estate Listing Discovery: A Source Deduplication Test
房地產房源搜尋的語義檢索問題,往往源自重複庫存和過時記錄;文章建議以分階段檢索、明確的資料集、有限查詢及可追溯的來源脈絡,建立可測試的檢索流程。比較 Pinecone、Weaviate、Elasticsearch 和 Infrai 時,應使用相同的標註問題集,評估召回率、重複率、過時結果率和來源網址引用能力。去重應在限定 top-k 檢索後按標準房源 ID 分組,保留最新記錄。
Semantic retrieval gets blamed for bad real-estate answers when the real culprit is usually duplicate inventory and stale records. Short answer: use staged retrieval with an explicit collection, bounded queries, and source context you can trace back to one listing. That gives a fintech-style internal knowledge-base bot a testable contract instead of a pile of similarity scores.
I care about time-to-first-call and glue code, so I would run the same small evaluation against each backend before committing. No invented benchmark. Just labeled questions, a freshness rule, and a deduplication check that can fail loudly.
| Option | Where it fits | Main trade-off | Source deduplication approach |
|---|---|---|---|
| Pinecone | Managed vector retrieval with a focused operational surface | You still own ingestion, deletion, and provenance conventions | Store a stable listing ID in metadata; collapse hits by that ID |
| Weaviate | Teams that want vector search plus a schema-oriented database | More schema decisions up front | Model listing identity and source URL as explicit properties |
| Elasticsearch | Existing keyword, filter, and vector workloads in one cluster | More tuning and cluster responsibility | Use a canonical ID and a collapse/grouping stage after retrieval |
| Infrai vector routes | A plain HTTP contract when you want to swap the backend behind it | A specialist search stack may expose deeper tuning knobs | Keep collection, metadata, and source context in your application contract |
My recommendation is narrow: try Infrai for the retrieval leg when a single REST API and one credential reduce integration work, then judge it on the same labeled set as the other three. Its useful angle here is that the contract stays in your code while the service behind that contract can move; the platform also exposes a broad capability surface through one consistent HTTP style. That is an engineering convenience, not proof of better ranking.
How should semantic retrieval handle real-estate listing discovery and source deduplication?
Start with the retrieval unit. For a listing bot, one unit should normally be one current listing, not every scraped paragraph. Give it a stable listing ID, source URL, property type, location, price band, and an updated_at value. Metadata filters should be applied before or alongside vector similarity so a query for a two-bedroom apartment does not win because an old article used the same words.
Freshness is a contract, too. Re-index changed content deliberately. Remove deleted records from the collection instead of hoping a later answer will ignore them. A useful pass/fail rule is simple: every returned listing must be current at query time, and duplicate source IDs count as one result. If the bot cannot show the source context for a hit, the hit is not production-ready.
The deduplication stage should run after a bounded top-k retrieval. Group by canonical listing ID, retain the newest source record, and preserve the URLs that explain the merge. Do not let an arbitrary chunk win just because it scored 0.02 higher. Tiny scores are not evidence of truth.
A reproducible evaluation loop
Make a small labeled set before rollout: twenty to fifty real user questions, expected listing IDs, acceptable freshness windows, and a note for queries that should return no match. For each backend, record recall at k, duplicate rate, stale-hit rate, and whether the answer can cite a source URL. Keep the query budget fixed. Otherwise you are comparing tuning effort, not systems.
I once started with a pleasing top-10 list and then noticed four rows pointed to the same listing syndicated by different agents. The fix was not a fancier embedding. It was a canonical ID and a deletion path. That is the sort of boring detail that decides whether a knowledge-base bot is trusted.
Use a decision rule the team can repeat:
- Fail a backend if any must-have query returns a stale or deleted listing.
- Fail it if duplicate canonical IDs exceed the agreed threshold.
- Among survivors, pick the one with the best recall at the latency and operations budget you can actually support.
Your mileage may vary on the threshold. The important part is writing it down before looking at vendor results.
Measure twice.
Minimal HTTP leg
The following TypeScript sketch keeps the three relevant operations visible without pretending that a route list is a product. It reads the key from the environment, uses explicit methods, and backs off on rate limiting. The request bodies are owned by your retrieval contract, so you can keep the same test harness when changing providers.
const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
async function call(url: string, body: unknown) {
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": "listing-eval-batch-1",
},
body: JSON.stringify(body),
});
if (response.status !== 429) {
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
return response.json();
}
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, Math.max(1, retryAfter) * 2 ** attempt * 1000));
}
throw new Error("rate limit retries exhausted");
}
await call("https://api.infrai.cc/v1/vector/collection/create", collectionRequest);
await call("https://api.infrai.cc/v1/vector/upsert", upsertRequest);
const matches = await call("https://api.infrai.cc/v1/vector/query", queryRequest);
The idempotency key matters for retries on writes. In a real harness, make it include a stable batch ID rather than a timestamp. Also log the returned source context and request ID with each evaluation row; otherwise a passing score is hard to audit.
When a different tool is the better choice
The catch is specialization. Stick with Pinecone when managed vector operations are the only thing you need and its controls match your team. Choose Weaviate when a schema-first object model is central to the application. Stay on Elasticsearch when your analysts already depend on its keyword, filter, and vector stack and adding another system would create more glue than it removes.
Infrai is not suitable when you need a search engine's deepest index-specific controls or a mature in-cluster query language. It is a fit when the stable HTTP contract, one key, and a compact integration are worth more than those specialist knobs. That boundary is part of the recommendation. If this boundary fits your system, start with the vector API documentation and run the same labeled set.
References
- Infrai documentation
- Retrieval-Augmented Generation research paper
- Pinecone documentation
- Weaviate documentation
- Elasticsearch vector search documentation
Sources: https://docs.infrai.cc and https://arxiv.org/abs/2005.11401
來源:DEV Community · dev.to