評審依循參照資訊:工具呼叫參數驗證中的推導差距
Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments
研究發現,工具呼叫評審模型在輸入缺少參數值參照資訊時,較少接受正確值,這種落差稱為「推導差距」。DeRef 以另一模型在不查看候選值的情況下讀取完整歷史並為評審撰寫參照;在三個基準測試中,它提升所有受測評審的整體參數 AUROC。受控測試中,AUROC 從隨機水平或以下升至 98.40 至 99.59。
Published on Oct 13
Authors:
,
,
,
Abstract
Tool-call judges evaluate argument values proposed by LLM agents. We find that they accept a correct value less often when their input holds no reference for it, that is, no text that says what the value should be. This happens when the value is a default value from the function schema that the user did not mention, when it has to be inferred, or when the text it comes from was cut from the judge input. We call this drop in acceptance the derivation gap. It appears across all seven judges we evaluate, including System One models CLM and Jev. The gap is not only a truncation artifact: correct default values have the largest gap on a benchmark with no truncation. A controlled source-relocation test shows that judges follow the reference in their input: they favor a visible incorrect value over a correct value whose source is hidden. We introduce DeRef (Derived Reference): a separate model reads the full history without seeing the candidate and writes a reference for the judge. Across three benchmarks, DeRef improves overall argument AUROC for all judges tested. On the controlled test, it raises AUROC from chance or below to between 98.40 and 99.59.
View arXiv page View PDF GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.18497
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.18497 in a model README.md to link it from this page.
Datasets citing this paper 1
lm2445/DeRef
Viewer • Updated about 4 hours ago • 8.29k • 44
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.18497 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co