研究發現:正確標籤仍不足,可靠元數據可悄然左右大語言模型行為
Correct Labels Are Not Enough: Reliable Metadata Can Silently Control Model Behavior
研究發現,Qwen3、Llama 和 Gemma 即使訓練標籤完全正確,仍可能在後訓練中學會信任無關元數據。把可靠的獎勵標註移到錯誤選項後,模型有 97–100% 的機率跟從錯誤答案;即使是「Florp」這類無意義標註也能產生相同影響。
Can perfectly correct training data still teach an LLM the wrong behavior? Surprisingly, yes.
We uncover a striking failure mode in LLM post-training: models can achieve high accuracy while silently learning to trust irrelevant metadata over the correct answer.
Across Qwen3, Llama, and Gemma, a perfectly reliable reward annotation becomes so influential that simply moving it to an incorrect option causes models to follow the wrong answer 97–100% of the time.
The surprising part? Every training label was correct. No malicious examples were needed. Even meaningless annotations such as "Florp" could acquire the same influence.
More concerning, removing the annotation restores most of the models' accuracy—meaning standard evaluations can completely miss this hidden dependency.
We also find evidence of similar cue dependence after preference optimization (DPO).
The takeaway: Correct labels do not guarantee correct learned behavior. Our findings reveal a subtle shortcut-learning risk in post-training and motivate counterfactual audits of reward scores, quality labels, and other training metadata.
What else might our models be learning to trust without us noticing?
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co