跳到正文
DEV Community· AI-KSK·· 5 天前AI 評分58

ComfyUI 聲音素材工作流程:先用 Stable Audio 3 生成背景音樂,再用 LTX 2.3 製作音訊響應動畫

Sound Assets in ComfyUI: Stable Audio 3 for BGM, Then LTX 2.3 for Audio-Reactive Animation

AI 導讀

作者分享兩套 ComfyUI 工作流程,先用 Stable Audio 3 生成音訊,再以音訊為條件用 LTX 2.3 生成響應動畫,讓音樂節奏與畫面動態由共同的時間來源連結。

正文

Most video pipelines treat audio as something you add afterwards. The two graphs here do the opposite: generate the sound first, then let the picture react to it. It is a small structural change with a large effect on how the result feels.

Graph 1: Stable Audio 3 asset generation

The small one — 24 nodes — and it is pure audio:

Node Value
EmptyLatentAudio [60, 1] — 60 latent frames
CLIPLoader t5gemma_b_b_ul2.safetensors, type stable_audio
SaveAudioMP3 audio/stable_audio_3, quality V0

That is the whole idea: an audio latent of a given length, conditioned through a T5-Gemma text encoder, decoded to MP3.

Two things are worth knowing:

V0 is the right export setting. Stable Audio output is already lossy-reconstruction territory; exporting at a lower MP3 bitrate stacks a second loss on top of the first, and the artifact shows up as a metallic edge on sustained tones.

60 latent frames is the length unit. Changing the sound's duration means changing that number, not trimming afterwards — trimming a generated piece leaves the tail un-decomposed and the music loses its ending.

Graph 2: audio-reactive animation on LTX 2.3

113 nodes. The audio path is not the interesting part — the interesting part is what the picture does with it:

  • LTX-2.3-OmniNFT-RL-Lora_bf16.safetensors at weight 1.2
  • LTXVAudioVAELoader → ltx-2.3-22b-dev-dare-ties-distilled-1.1.safetensors
  • EmptyLTXVLatentVideo at 320×288, 97 frames
  • LTXVEmptyLatentAudio at 97, 25, 1
  • KSamplerSelect → euler_ancestral_cfg_pp

The LoRA weight of 1.2 is above 1.0, which is unusual and deliberate. OmniNFT-RL is a reinforcement-tuned LoRA; pushing past 1.0 amplifies the reactivity it learned. At 1.0 the motion follows audio cues but conservatively — at 1.2 the linkage is visible in the frame, not just inferable.

The cost of 1.2 is the usual one: motion that reacts strongly to audio also reacts to audio artifacts. A click in the track becomes a visible hitch.

Why do audio first

If you generate picture first and add music later, the two are unrelated — you are synchronising by hand or accepting that they ignore each other.

Generating audio first and conditioning picture on it means the motion shares a source with the sound. Beats land on motion peaks because both derive from the same latent timing, not because someone cut it that way.

That is the actual claim of this pairing, and it is checkable: mute the audio and the motion still reads as rhythmic. That only happens when the picture was conditioned on the audio rather than cut to it.

Where the pipeline breaks

  • Silence in the track — an audio-reactive graph has nothing to react to. Pad with room tone, do not leave true silence.
  • Sharp transients — clicks, pops and hard cuts in the track produce frame-level hitches. The graph amplifies what the audio says, including mistakes.
  • Length mismatch — the audio latent and the video latent are separately specified (60 frames vs 97 frames in these two graphs). Reconciling them means choosing one as the master and deriving the other, not generating both independently.

Practical pairing

Generate the audio at the length you need, inspect it on its own before feeding it to the video graph, and only then run the animation. Reactive motion is not a way to fix bad audio — it is a way to make good audio visible.

Workflows

Two graphs, published on RunningHub with invite code rh-v1111:

  • Stable Audio 3 sound asset generation
  • LTX 2.3 audio-reactive animation

Files

The ComfyUI workflow exports are in the companion repository:

https://huggingface.co/AI-KSK/ksk-video-249-comfyui-workflows

Video walkthrough (Chinese): https://youtu.be/437DDTcszfs
Bilibili: https://www.bilibili.com/video/BV1gLGo6kEbF/

Notes

  • Workflow files only. Model weights are not included; each model keeps its own licence.
  • These are tutorial exports, not a packaged product.
  • Node counts and widget values quoted above are read from the published JSON files.
  • Independent AI-KSK release. Not affiliated with or endorsed by Stability AI, Lightricks or RunningHub.

來源:DEV Community · dev.to