跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 18 天前AI 評分39

CARTBench:以評分準則評測附時間線的複雜及含糊購物需求建議

CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines

AI 導讀

CARTBench 評估大型語言模型對附時間線的複雜及含糊購物需求所提供的建議。基準涵蓋 17 個零售領域、1,603 條產品紀錄及 1,328 項專家標註準則。評測包括 7 款商用 API 模型(包括 GPT-6.1-Sol 和 Claude-Opus-5)及 35B+ 開放權重模型;10 款模型中,3 款 Core 得分超過 90%,沒有模型的 Excellence 得分超過 45%。

正文

Published on Nov 4

Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines). For Data scarcity, we curate cases from real commercial shopping needs across 17 retail domains, pairing timestamped dialogue with 1,603 product records. For Difficulty Standardizing Criteria, we develop case-specific rubrics guided by Ideal Response Principles, distilled from excellent replies manually selected from successfully completed orders. These principles also guide Principle-Guided CART Prompting, our structured recommendation strategy. The experimental benchmark contains 1,328 expert-annotated criteria across Core, Excellence, and Safety. To answer the CART question, we evaluate 7 flagship commercial API models, including GPT-6.1-Sol and Claude-Opus-5, and 35B+ parameters open-weight models. In the first full-suite evaluation under the frozen rubrics, 3 of 10 models exceed 90% on Core, while none exceeds 45% on Excellence. Therefore, we identify an understanding-to-execution gap between required behavior and high-quality purchase advice. To examine these limitations, we diagnose shared failures in product evidence and unresolved needs, linked to changing requirements and decisive unknowns. Code and data are available at https://github.com/Da1yuqin/CARTBench.

View arXiv page View PDF GitHub Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.05862 in a model README.md to link it from this page.

Datasets citing this paper 1

dayll/CARTBench

Viewer •

Updated about 1 hour ago

•

169

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.05862 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co