MultiAgent-OPD:以 On-Policy Distillation 端到端訓練階層式多智能體系統
MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation
MultiAgent-OPD 以 On-Policy Distillation 端到端訓練階層式多智能體系統,結合 Searcher 的局部過程監督,以及 Leader 和 Answerer 基於結果的強化學習。
✨ Highlights
Role-specific learning signals — MultiAgent-OPD combines local process supervision for Searchers with outcome-based reinforcement learning for the Leader and Answerer, addressing credit assignment in hierarchical multi-agent teams.
Two-stage training recipe — First trains a teacher to search and summarize in a cascaded Search–Answer workflow, then transfers its expertise to Searchers during joint hierarchical training.
Token-level on-policy distillation — A frozen, task-trained teacher scores student-generated queries and summaries on evolving subtask trajectories, providing fine-grained guidance for evidence acquisition and reporting.
Joint training with a shared model — Leader, Searcher, and Answerer roles jointly optimize a shared language model to coordinate task decomposition, multi-wave search, and answer generation. Inference requires only the trained student.
Strong performance across seven QA benchmarks — MultiAgent-OPD achieves 38.68% macro avg@4, outperforming M-GRPO like LLM Judge MAS-RL method by 4.06 percentage points and C-MAS + RL (cascaded MAS-RL method) by 2.61 points at the evaluated checkpoints.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co