NitroFusion2 將對抗式損失引入多模態 Diffusion Transformer 的文字流
NitroFusion2: Bringing Adversarial Loss to the Text Stream of Multimodal Diffusion Transformers
NitroFusion2 在多模態 Diffusion Transformer 中,使用凍結 MMDiT 的中間文字 token 狀態作為判別器表示,提出 Text-Stream Adversarial Distillation。
Hi, first author here.
In an MMDiT, text and image tokens share every attention layer, so by mid-network the text tokens have already read the image. We use these intermediate text-token states of a frozen MMDiT as the discriminator's representation (Text-Stream Adversarial Distillation). In controlled ablations, text-stream discrimination outperforms conventional image-stream discrimination, and combining both streams is best on most metrics.
Because the discriminator reads the text stream, its text condition also steers what it judges: swapping the cue (typography, objects, style, …) turns it into an expert on that aspect, with aspect-specific gains (Aspect-Cued Discrimination).
Built on Z-Image-Turbo, NitroFusion2 brings few-step distillation from 4–8 steps down to 1–2 steps.
Questions and feedback welcome!
If you find NitroFusion2 interesting or useful, an upvote here or a star on GitHub helps more people find it. Thank you🙏
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co