跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-16AI 評分37

大型語言模型邏輯推理的能力本位評估

Competence-Based Evaluation of Logical Reasoning in Large Language Models

AI 導讀

研究提出能力本位評估(CBE),檢驗大型語言模型能否在不同問題表述及複雜度下穩定運用同一推理能力。在排序任務中測試六個 GPT 和 Qwen 模型,結果顯示模型雖在有限複雜度範圍內表現不俗,仍未能在整個測試系列中持續符合能力條件。Chain-of-thought 和 GRPO 未能解決所有測試失誤,監督式 LoRA 的留出遷移效果有限。

正文

Published on Oct 16

Authors:

Abstract

Competence-based evaluation reveals that large language models fail to consistently apply reasoning rules across varied problem presentations despite high accuracy on bounded tasks.

Recent advances in large language models (LLMs) have brought impressive performance across diverse tasks; however, shortcomings remain in logical reasoning, including inconsistencies under linguistic variation. To improve the reasoning capacity of LLMs, we must be able to accurately measure their reasoning capacity. Current benchmarks primarily report single accuracy numbers on selected reasoning tasks. Such a performance-based evaluation methodology says little about a model's generalization capacity in reasoning. To address this limitation, we argue for a different evaluation paradigm, called competence-based evaluation (CBE), which tests whether models reliably apply the same competence across a family of problems. A parameterized generator scales complexity while keeping the required competence fixed, and matched answer-preserving presentations test correctness beyond one formulation. We instantiate CBE on elementary ordering tasks, scaling input size while varying presentations of the same ordering rule. Across six GPT and Qwen models, performance is reasonably good over bounded complexity ranges, but even strong models fail the competence condition across the tested family. Chain-of-thought and GRPO extend strong performance without resolving all tested failures; supervised LoRA shows little held-out transfer. CBE provides diagnostic profiles and an evaluation protocol that can grow with future models.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.22894

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.22894 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.22894 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.22894 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co