跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 2026-10-14AI 評分52

CALIF 診斷並修正可驗證指令遵循中的獎勵駭客行為

Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

AI 導讀

研究提出 CALIF,以校準似然獎勵處理可驗證指令遵循中,輸出滿足限制卻未實質回答請求所造成的獎勵駭客問題。CALIF 根據請求本身,在凍結參考策略下評估回應似然,毋須額外評審或學習式獎勵模型。

正文

Published on Oct 14

Authors:

,

,

,

,

,

,

Abstract

Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2610.19525

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.19525 in a model README.md to link it from this page.

Datasets citing this paper 1

sangyon/CALIF

Updated about 1 hour ago

•

14

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.19525 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co