Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders
Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders
ReMaRk 針對 LLM 推薦系統提出以請求而非單一回答為單位的強化學習訓練方法,將所有回答中的標題合併評分,並讓每個請求獲得相同抽樣權重。測試涵蓋七個骨幹模型和三個數據集;ReMaRk 以 25 至 53% 的訓練步數,達到 Rank-GRPO 經 2,000 步訓練後的驗證集召回率。
Published on Oct 13
·
on Oct 15
Authors:
,
,
,
Abstract
A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is. We argue that RL should train on the request, not on each answer, and propose ReMaRk: it scores every list once against the union of all replies' titles and samples each request equally. This makes RL cheaper. On seven backbones and three datasets, ReMaRk reaches the validation recall of 2,000 per-reply RL steps in 25 to 53% of the steps and leads at that budget; trained until neither improves, it is never significantly behind Rank-GRPO and in four of seven pairs reaches Rank-GRPO's best with fewer GPU-hours. The same reward helps on ProtoQA, where about 100 people answer each question, which points to a simple rule for RL with several valid human answers: reward them jointly.
View arXiv page View PDF Project page GitHub Add to collection
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co