Soundify:為影片配對音效
Soundify: Matching sound effects to video
Soundify 是一套為影片配對音效的系統,透過帶標籤的錄音室級音效庫,並將 CLIP 擴展為「zero-shot detector」,產生高質素結果。系統無須耗費大量資源進行音視覺對應學習或音訊生成;研究人員訪問 10 位專業影片剪輯師後,發現尋找合適音效、精確配對影片及逐格調校參數都十分耗時。
Authors
David Chuan-En Lin1, Anastasis Germanidis2, Cristóbal Valenzuela2, Yining Shi2, Nikolas Martelaro1
In the art of video editing, sound is really half the story. A skilled video editor overlays sounds, such as effects and ambients, over footage to add character to an object or immerse the viewer within a space [1]. However, through formative interviews with 10 professional video editors, we found that this process can be extremely tedious and time-consuming. More specifically, video editors identified three key bottlenecks: (1) finding suitable sounds, (2) precisely aligning sounds to video, and (3) tuning parameters such as pan and gain frame-by-frame. To address these challenges, we introduce Soundify, a system that matches sound effects to video. Prior works have largely explored either learning audio-visual correspondence from large-scale data [3, 9, 8] or performing audio synthesis from scratch [5, 10, 4]. In this work, we take a different approach. By leveraging labeled, studioquality sound effects libraries [2] and extending CLIP [6], a neural network with impressive zero-shot image classification capabilities, into a "zero-shot detector", we are able to produce high-quality results without resource-intensive correspondence learning or audio generation.
Example Visualization
Example Results
來源:Runway:News(網頁) · runway.com