繼承而非創造:追溯推理模型中 {wait} 標記的來源
Inherited, Not Invented: Tracing the Provenance of Wait in Reasoning Models
研究追溯推理模型中 {wait} 標記的來源,發現強化學習提升推理準確度並不需要這個標記;標記主要源自訓練資料,而非強化學習。基礎模型原本沒有 {wait} 時,零強化學習幾乎不會產生它,但加入強化學習後 MATH-500 準確度仍大幅提升。研究亦指出,單看標記頻率不足以判斷推理變化,自動評審的修訂指標可能與人工標註趨勢相反。
Published on Oct 22
Authors:
,
,
,
,
,
,
,
Abstract
Reinforcement learning improves reasoning accuracy without requiring a specific wait marker, which originates from training data rather than RL, and automated evaluation metrics can misrepresent reasoning trends.
Reinforcement learning (RL) is credited with teaching reasoning models to write {wait} before reconsidering a step, yet large RL accuracy gains do not require the word. This view can confuse what RL discovers with what a model inherits from its training data. We trace the word through open models' training histories, reading it with label-free indices of frequency, position, form and usage and an injected-error test, all computed from sampled outputs. We find that zero-RL almost never coins the marker: on bases that lack the word, RL raises MATH-500 accuracy substantially while {wait} stays near absent. It arrives with training data: mid-training in OLMo-3, and fine-tuning on reasoning traces installs it in most models whose base lacks it. Where the base already uses it, both released RL-Zero recipes (OLMo-3, MiMo) multiply it, whereas KL-free GRPO from the same base does not; OLMo-3's recipe wears down its form and makes it a more sensitive but more liberal alarm; RL after supervised fine-tuning barely changes it, and our GRPO recipe erases a marker seeded by fine-tuning. Finally, we show that marker frequency alone does not indicate RL-induced reasoning and caution against LLM-as-a-judge revision metrics, as automated judges can reverse human-labelled trends.
View arXiv page View PDF GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.30882
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.30882 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.30882 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.30882 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co