M3OPD-Reasoner:以多教師 on-policy 蒸餾進行多模態推理後訓練的開放方案
M3OPD-Reasoner: An Open Recipe for Post-Training Multimodal Reasoning with Multi-Teacher On-Policy Distillation
M3OPD-Reasoner提出一套完全開放的多模態推理後訓練方案,涵蓋圖像 STEM、圖像感知及影片三個領域。評測 21 個基準後,研究發現從 Qwen3.5-9B 直接以 GSPO 訓練各領域教師,再用 MOPD 蒸餾的方案表現最佳;接續 MixRL 仍可提升學生模型,形成 M3OPD-Reasoner-9B。
Published on Nov 17
Authors:
,
,
,
Abstract
Multi-Teacher On-Policy Distillation (MOPD) enables a single student to absorb the skills of several domain teachers by learning from its own generated responses. We present Multimodal Model Multi-Teacher On-Policy Distillation Reasoner (M3OPD-Reasoner), a fully open recipe for post-training vision-language models (VLMs) over three reasoning domains: image STEM, image perception, and video. We curate two datasets, M3OPD-SFT-1.35M for supervised fine-tuning (SFT) and M3OPD-RL-193K for reinforcement learning (RL) with verifiable rewards. On these data, we compare post-training recipes that start from Qwen3.5-9B: training one model on the mixed data of all domains (MixSFT and MixRL), and training one teacher per domain with Group Sequence Policy Optimization (GSPO), directly or after an SFT warm-up, and distilling the teachers into one student with MOPD. We evaluate on 21 image STEM, image perception, and video reasoning benchmarks after we decontaminate the training data against them. The comparison establishes MOPD of teachers trained with GSPO directly from Qwen3.5-9B as the best recipe: the student closely matches each teacher on its own domain and is on average above the mixed-training baselines on the same prompts. Since MOPD already merges the domain capabilities that MixRL would otherwise have to learn, its student is a stronger starting point for RL, and we ask whether the same data can carry the model further from there: we add MixRL on top of the MOPD student, on the same prompts and with the same settings, which, to our knowledge, no prior VLM recipe has studied. The same prompts still improve the MOPD student, with no collapse, and give our main model, M3OPD-Reasoner-9B. MOPD and MixRL can therefore work as consecutive stages that get more out of the same data. We release all data, the training code, and the checkpoints of every stage.
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2611.23034
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 16
Image-Text-to-Text • 9B • Updated about 1 hour ago • 1ngqtrung/M3OPD-MixSFT-9B
Image-Text-to-Text • 9B • Updated about 1 hour ago • 46ngqtrung/M3OPD-Warmup-SFT-Video-9B
Image-Text-to-Text • 9B • Updated about 1 hour ago • 38ngqtrung/M3OPD-Teacher-ImageSTEM-9B
Image-Text-to-Text • 9B • Updated about 1 hour ago • 36ngqtrung/M3OPD-Teacher-ImagePerception-9B
Datasets citing this paper 2
ngqtrung/M3OPD-SFT-1.35M
Viewer • Updated about 1 hour ago • 1.35M • 120
ngqtrung/M3OPD-RL-193K
Viewer • Updated about 1 hour ago • 193k • 116
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.23034 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co