跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 5 天前AI 評分49

LLM 購物智能體想得更多,選擇不會更好

Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents

AI 導讀

在三個購物基準測試中,提高 LLM 購物智能體的思考力度,未能在統計上改善產品選擇質素,卻顯著提高其自我報告的信心。大多數模型在不同思考力度下校準不佳,而且常在相同任務中出錯,甚至選中同一件錯誤產品。

正文

Published on Nov 17

Authors:

Abstract

Across three shopping benchmarks, increasing large language model shopping agents' thinking effort did not statistically improve product-choice quality but significantly raised reported confidence, and most models were poorly calibrated with correlated errors.

Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.

View arXiv page View PDF Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.22033 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.22033 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.22033 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co