跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 16 天前AI 評分40

LambdaMix:以聯合訓練低秩適配器實現可調式多目標羣組相對偏好最佳化

Steerable Multi-Objective Group-Relative Preference Optimization with Jointly Trained Low-Rank Adapters

AI 導讀

LambdaMix 以羣組相對線上強化學習,在凍結的基礎模型上為每個目標聯合訓練一個低秩適配器,學習一系列由偏好條件化的策略。偏好向量會選擇生成回應的適配器組合,並加權用於更新適配器的獨立正規化獎勵;訓練完成後,可在推理時調整偏好而無須再次最佳化。LambdaMix 在四個基礎 LLM 上的超體積高於所有受評估基線,所需訓練補全數量及適配器儲存空間亦少於一組 GRPO 專家模型。

正文

🤔 Post-training usually gives you one policy for one fixed weighting of objectives, but different users and settings call for different trade-offs.

🎛️ We introduce LambdaMix, a parameter-efficient method that learns a whole family of preference-conditioned policies with group-relative online RL.

🧩 LambdaMix trains one low-rank adapter per objective together on a frozen base model. A single preference vector does two things: it picks the adapter mixture that generates responses, and it weights the separately normalized rewards used to update the adapters.

🔁 After one training run, you can change preferences at inference with no further optimization.

We test LambdaMix across four base LLMs:

  • 🛡️ on helpfulness vs. harmlessness
  • 🧮 on math with instruction following
  • 🕵️ on rewriting comments while removing clues about personal attributes

📈 LambdaMix achieves higher hypervolume than all evaluated conditioned and policy-merging baselines.

⚡ It also uses fewer training completions and less adapter storage than a set of GRPO specialists.

💻 Code: https://github.com/ckarouzos/lambdamix

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co