AutoVend:長期自主自動販賣營運中的 LLM Agent 評測基準
AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
AutoVend 是一項評測 LLM Agent 長期自主營運自動販賣機的基準,模擬一個 Agent 透過網上介面管理五種地點的 20 部自動販賣機,為期 180 日。五個受測 LLM Agent 中有四個表現不及規則式庫存控制基準,按推理成本計算的回報亦會改變模型排名。分析顯示,庫存與執行成本會侵蝕個別指標的增益,Agent 亦難以制定可行行動及根據回饋有效修訂策略。
Published on Oct 27
Authors:
,
,
,
,
,
,
,
,
Abstract
Large language model (LLM) agents are extending agent-mediated commerce on the Web from shopping for consumers to operating businesses for sellers. In online-to-offline (O2O) commerce, decisions are issued through Web-based platforms but realized through costly physical execution. Evaluating such agents requires going beyond individual transactions to ask whether decisions made online turn into sustained economic returns. We introduce AutoVend, a benchmark in which one LLM agent manages 20 vending machines across five site types through a Web-based operating interface for 180 simulated days. Its data-grounded simulator combines a shared warehouse, procurement lead times, paid replenishment, inventory expiration, and recurring operating costs. We evaluate five LLM agents against a rule-based inventory-control baseline and diagnose their site coordination, execution, and performance over time. Four of five evaluated LLM agents fall short of this baseline, and returns per dollar of inference cost rank models differently from returns alone. Operational analyses show that gains in individual metrics do not consistently yield stronger economic growth, as inventory and execution costs erode these gains. Trajectory analysis further reveals difficulties in grounding feasible actions, managing inventory commitments, and converting feedback into effective policy revisions.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2610.36980
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.36980 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.36980 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.36980 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co