跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-14AI 評分42

RuleTrek:複雜情境中智能體規則推理基準評測

RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios

AI 導讀

論文提出 RuleTrek,作為評測複雜情境中智能體規則推理能力的多領域基準。在 11 個前沿大語言模型(LLM)的測試中,規則識別(RI)和規則推導(RD)最高分分別只有 62.47% 和 60.43%,端到端規則推理(E2E)得分介乎 26.97% 至 55.29%。程式碼及數據集已公開提供。

正文

Published on Oct 14

Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introduces RuleTrek, a multi-domain benchmark for agentic rule reasoning in complex scenarios. Its rule-grounded construction pipeline extracts structured rule graphs, generates and incrementally composes rule-specific sub-scenarios while ensuring consistency and solvability. To systematically evaluate agentic rule reasoning in complex scenarios, RuleTrek assesses three capabilities in interactive workspaces: Rule Identification (RI), Rule Deduction (RD), and End-to-End Rule Reasoning (E2E). Experiments on 11 frontier LLMs show that the highest RI and RD scores are only 62.47\% and 60.43\%, respectively, while E2E scores range from 26.97\% to 55.29\%. These results demonstrate that current LLM agents face substantial challenges in rule reasoning over complex scenarios. We make our code and dataset publicly available at https://github.com/yuan247/RuleTrek{RuleTrek}.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.19698

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.19698 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.19698 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.19698 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co