COSMI: COmpositional Synthesis of Multi-object Interactions
COSMI: COmpositional Synthesis of Multi-object Interactions
COSMI結合由單物體捕捉組成的數據集與文字轉交互模型,用於生成多物體人與物件交互。數據集包含222k段序列、275小時的數據,最多涵蓋5件物件,規模接近最大的已捕捉多物體數據集的30倍。30M參數的擴散 Transformer 在基準測試中於文字對齊和接觸準確度領先,尤其在未見物件上;程式碼、模型和數據集流程將會發布。
🧩 COSMI — multi-object human-object interaction in two parts: a dataset composed from single-object captures, and a text-to-interaction model trained on it.
📦 Data: interactions are local (a hand holds a cup, a chair supports the pelvis), so a limb and its object transfer between bodies. An LLM and geometric checks keep only plausible pairings, and mirroring balances left and right hands (49/51). The dataset has 222k sequences / 275 hours with up to 5 objects, nearly 30× the largest captured multi-object dataset.
🧠 Model: one 30M-parameter diffusion transformer generates interactions with 1 to 5 objects from text. Object slots share weights, and each object is predicted relative to the body part that moves it.
📊 Benchmark: holds out an unseen object and 5 unseen interaction combinations, such as walking with an apple. All models trained on the dataset generalize to the unseen combinations. COSMI leads in text alignment and contact accuracy, most clearly on the unseen object.
📈 The dataset grows by adding new datasets, even hand-object recordings (shown with HOT3D). Code, models and the dataset pipeline will be released.
🔗 Paper · Project page · GitHub · Hugging Face
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co