嵌入向量與 grep:重新審視程式碼搜尋中的稠密檢索優勢
Embeddings vs. Grep: Rethinking the Dense Advantage in Code Search
這項研究提出 Repo-Premium,一個以真實程式碼庫為基礎的程式碼搜尋基準,用來重新檢視稠密檢索相對 BM25 的優勢。研究發現,評測設計偏向稠密檢索及查詢缺少程式碼中的稀有識別符號,是優勢的兩個來源;加入由 LLM 生成的識別符號後,BM25 在每個 Python 測試池均追平稠密檢索。
Published on Nov 16
Authors:
,
,
,
,
Abstract
Code-search benchmarks show dense encoders well ahead of BM25, yet coding agents do well with grep. What does this dense advantage measure? Existing benchmarks cannot tell: they use curated queries, old code and a handicapped BM25. To find out, we build Repo-Premium, a realistic repository-level code-search benchmark. It ranks files in real repositories, with curated and raw queries in Python, JavaScript, TypeScript, Go, Rust and Java, pull requests from 2022 to 2026, and a preregistered test pool over 91 repositories. Our experiments trace the advantage to two sources: evaluation choices that favor dense retrieval, such as splitting files into chunks for BM25 and counting edited tests as relevant, and queries that lack the code's rare identifiers. Expanding the query with LLM-written identifiers (generate-then-grep) helps BM25 far more than dense retrieval, letting it match dense retrieval in every Python pool, even on pull requests opened after the LLM was released. No method we tried predicts which retriever will win on a given query. From these results we derive three guidelines: measure dense gains against a fair lexical baseline, fuse dense retrieval with BM25, and add the code's words to queries that lack them.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2611.20455
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.20455 in a model README.md to link it from this page.
Datasets citing this paper 1
hq-bench/repo-premium
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.20455 in a Space README.md to link it from this page.
Collections including this paper 2
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co