SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation
SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation
SyncReward 研究視聽生成的同步評估與獎勵建模,並推出配套資料集、基準及獎勵模型。SyncReward-Data-80K 收錄 80K 項專業同步評分,SyncReward-Bench 包含 1K 段由多位專業人士獨立評分的片段;SyncReward 與受評方法相比,最能貼合人類評分。作為獎勵模型,它可透過推理時 best-of-N 選擇及基於強化學習的後訓練改善同步效果。
Published on Oct 12
Authors:
,
,
,
,
,
Abstract
Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio--visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct SyncReward-Bench, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-N selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code at https://github.com/WeChatCV/SyncReward.
View arXiv page View PDF GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.16338
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.16338 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.16338 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.16338 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co