WORTH Benchmark:非完全合作資訊下的證據發現——兩集試點研究
WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot
WORTH Benchmark 評估模型在懸疑故事中自主發現證據及推理的能力,兩集試點顯示線索選取與人物判斷並不同步。Episode 02 的 37 次可評分測試中,31 次選出相同的一組錯誤人物;疑點句與洗脱嫌疑句分別被選取 25 次和 0 次。研究僅涵蓋兩集,未建立穩定的模型排名,也未報告人類與模型的比較。
Published on Oct 20
·
on Oct 26
Authors:
Abstract
The WORTH Benchmark evaluates autonomous evidence discovery and reasoning in mystery narratives, revealing that models select clues and participants inconsistently due to narrative structure.
Existing reasoning evaluations typically provide task-relevant information to models as available premises, rarely examining concurrently how models discover evidence autonomously and how information presentation within narratives guides or obscures judgment. This paper proposes the WORTH Benchmark: models read original mystery narratives without clue annotations, autonomously select key clue sentences, identify case participants, and explain their reasoning process. The narrative materials are designed by professional authors such that evidence facilitating reasoning coexists with narrative arrangements that delay the revelation of truth. We report a pilot experiment under full-story conditions across two episodes, conducting a total of 70 formal runs on models, of which 67 runs are content-scorable. Scoring separately records hits on author-designated key clue sentences and participant selections; natural language explanations are used for case diagnosis and are not scored independently. The results show that clue hits and participant judgments are not synchronized; across 37 content-scorable runs in Episode 02, 31 runs simultaneously selected the same set of incorrect participants, while adjacent suspicion-oriented and exoneration-oriented sentences were selected 25 times and 0 times, respectively. These phenomena illustrate the diagnostic value of component-wise analysis and suggest that not-fully-cooperative informational relationships may influence evidence selection and judgment. This study only implements full-story conditions for two episodes, and does not establish stable model rankings or infer general mechanisms on this basis; human player records have not yet been cleaned, and this version does not report human-model comparisons.
View arXiv page View PDF Project page Add to collection
Get this paper in your agent:
hf papers read 2610.27597
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.27597 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.27597 in a dataset README.md to link it from this page.
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co