我打造雙人 AI 嘻哈對嘴影片生成器的心得
What I Learned Building a Two-Person AI Lip-Sync Rap Video Generator
作者打造 AI Rap Video Generator,讓用戶上載兩張照片、免費編輯 AI 生成的歌詞,再製作約 12 秒的直向雙人對嘴嘻哈影片。產品的關鍵設計是先讓用戶修改歌詞,再進行付費渲染,並確保兩個人物在影片中輪流演唱時保持面貌一致。
Disclosure: I'm the maker of the product described here.
Most AI video tools I tried last year were built for one job: a single talking head reading an ad script. That's useful if you run a UGC agency. It's less useful if you just want to make your best friend rap a roast about being late to every dinner for their birthday.
That gap is where this side project started.
The problem
I wanted something small and silly: take two photos of real people (friends, a couple, siblings), give them a short rap, and get back a vertical video you could drop into a group chat or a story. No timeline editor, no avatar marketplace, no "enterprise seats."
The requirements turned out to be stricter than they sound:
- Two faces, one video. A duo has to look like a duo — trading lines, not two separate clips glued together.
- Rhythm matters. Rap exposes bad mouth timing instantly. A talking-head model that's "good enough" for narration looks off-beat on a verse.
- The words are the joke. If the lyrics are generic, the video is pointless. People want their inside jokes in there.
What we tried
Attempt 1: talking-photo animation. The first prototype used a generic "animate a photo to speech" approach. It nodded, it blinked, the mouth moved. But mouth movement is not lip-sync. On fast syllables the lips drifted, and on held notes the face kept "talking." For narration nobody notices; for rap, everybody does.
Attempt 2: generate lyrics, then render immediately. We let the AI write lyrics and went straight to render. Users hated it. The AI got names wrong, missed the actual joke, or wrote something too generic. Then they'd re-render, wait again, and (in a paid flow) pay again for something they never wanted.
Attempt 3: render each person separately, then composite. Simpler pipeline, but the two characters felt disconnected, and across segments a face would subtly change — "duo identity drift." Person A at second 2 didn't quite look like Person A at second 10.
What we shipped
The current flow of AI Rap Video Generator is deliberately short:
- Upload two photos.
- Review and edit the AI-generated lyrics — for free. Fix names, add the inside joke, cut the line that's too mean.
- Render a ~12-second lip-synced rap duo video, vertical, sized for social.
That's it. Twelve seconds is a choice, not a limitation we're embarrassed about: it's long enough for a punchline and short enough to watch twice in a group chat.
Lessons
1. Lip-sync and talking-photo are different products.
Talking-photo optimizes for "the face looks alive." Lip-sync optimizes for "the mouth matches these exact phonemes at these exact times." Music makes the difference obvious. If your audio has a beat, judge your model on syllable timing, not on how natural the head motion looks.
2. Let people edit before they pay.
Moving lyric editing before the render — and making it free — was the single best UX change. The expensive step (rendering) only happens once the user is happy with the cheap step (text). It reduces wasted compute, reduces refund-style frustration, and makes the output actually personal. General rule: put the editable, cheap artifact in front of the costly, irreversible one.
3. Duo identity drift is its own problem.
Single-face consistency is mostly solved. Two faces in one clip, trading lines, is harder: each identity has to stay stable across the whole video, and they must not bleed into each other. Treat identity consistency as a first-class test case — compare faces at start, middle, and end of every render, not just the first frame.
4. Niche positioning is a feature.
We're not trying to compete with enterprise talking-head UGC ad tools. Birthdays, roasts, friends, couples — that's the use case. Saying no to the enterprise feature list kept the product to three steps.
5. Short output, fast feedback.
A ~12-second format means people actually finish watching, share, and come back to make another. For a fun product, the loop matters more than length.
Wrapping up
If you're building anything with generated media, my biggest takeaway is: find the cheapest artifact the user cares about (here, the lyrics) and let them perfect it before you spend anything on the expensive one.
If you want to see how it turned out, you can try AI Rap Video Generator at https://www.airapvideogenerator.app/ — two photos, edit the lyrics, get a short rap duo video. Feedback (especially on lip-sync timing) is very welcome.
來源:DEV Community · dev.to