跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 4 天前AI 評分41

虛假的成功,無聲的失敗:審核模擬操作基準測試的成功檢查器

Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks

AI 導讀

SimAudit 透過大語言模型審核模擬操作基準測試的成功檢查器,找出檢查器與任務指示可能不一致的狀態,並在模擬環境中重現及核實。常用基準測試均出現檢查器錯誤,例如 RoboTwin 2.0 會同時誤判成功與失敗,因此公佈的成功率未必能準確反映執行情況。

正文

Are your robot policies really succeeding, or is the success checker lying? 🤖

Simulated manipulation benchmarks rely on hand-written success checkers, which are usually validated only against expert demos. Learned policies behave differently, so checkers can:

  • ✅ accept incomplete executions → phantom success
  • ❌ reject executions that actually follow the instruction → silent failure

We introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. It reads the instruction and checker code, proposes states where they may disagree, recreates them in simulation, and verifies the outcome with multi-view renderings.

🔍 Key finding: checker errors appear across popular benchmarks (e.g., RoboTwin 2.0 errs in both directions), so reported success rates may not mean what we think.

Feedback welcome, especially from people building or using manipulation benchmarks!

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co