Hu Jin 如何製作角色一致的 90 秒 AI 短片,並用 ffmpeg 修正瑕疵
How I Made a 90-Second AI Short Film With One Consistent Character (and Fixed Its Glitches With ffmpeg)
Hu Jin 以 AI 影片工具製作約 90 秒短片《Mei 2036》,並分享利用 ffmpeg 檢查及修正生成瑕疵的流程。他以兩張角色參考圖、固定描述和每個鏡頭的首幀圖維持人物一致,並將片段抽取至每秒 4 格逐格檢查。文章亦提供簡化人物動作、剪短問題片段及以腳本串接影片的做法;最終剪輯長約 1:27。
I'm a software and data engineer, not a filmmaker. A few weeks ago I wanted to find out how far one person can get with today's AI video tools if they treat a short film like an engineering project: a spec, reproducible inputs, a QC pass and a build script.
The result is Mei 2036, a short film about an ordinary morning ten years from now. A 22-year-old named Mei wakes up, her mirror checks her health, a robot arm makes breakfast, a floating assistant reminds her about a meeting, a driverless shuttle takes her through a quiet green city, she works with a robot in a design studio, and the film ends on the skyline. Eight shots, one character, around 90 seconds.
This post covers the whole pipeline, including the parts that went wrong. If you want to make your own AI short, you should be able to reuse most of it.
TL;DR
- Lock the character first. Two reference sheets (day clothes and pajamas) and a fixed text description, pasted into every prompt.
- Generate from a still, not from text. Each shot starts from its own first-frame image (image-to-video). That is what keeps the face the same.
- Simple motion beats clever motion. Every bad glitch I hit came from hands, props or bodies touching objects.
- QC frame by frame. Extract several frames per second with ffmpeg and look at them. Watching at full speed hides a lot.
- Treat audio as its own build. I muted the original speech, generated English voiceover with neural TTS, synthesized my own music and mixed it with sidechain ducking to about -16 LUFS.
-
One script stitches everything.
ffmpegwithxfadefor transitions anddrawtextfor the end title. - The final cut plays at 1.25x speed (about 1:27). The GIFs in this post come from that version.
1. Concept and an 8-shot storyboard
I wanted a story that is small, warm and easy to read without dialogue: the future isn't colder, it just wakes up a little earlier. One character, one morning, one location per shot, and nothing that needs complicated acting.
I wrote the storyboard as a table before generating anything. Each row had the shot length, the setting, what happens, the motion prompt, the voice line and the sound.
| # | Shot | Length | What happens |
|---|---|---|---|
| 1 | Wake up (06:30) | 15 s | Smart window clears, sunlight, Mei stretches, the bedside orb glows |
| 2 | Mirror check | 12 s | The mirror shows sleep, heart rate and a hydration reminder |
| 3 | Breakfast | 13.5 s | A robot arm serves breakfast. Tomatoes grow in an indoor garden |
| 4 | AI assistant | 15 s | A floating orb reads out her schedule |
| 5 | Leave home | 12 s | She walks out toward a driverless pod shuttle |
| 6 | Shuttle window | 15 s | A quiet green city drifts past |
| 7 | Studio | 15 s | She works on a hologram with colleagues and a robot |
| 8 | The view | 15 s | Floor-to-ceiling window, the end title fades in |
Here is the first frame of every shot. These eight images are the most important files in the project:
Tip: keep each shot to one idea and at most one action. "She walks to the shuttle" works. "She walks to the shuttle, opens the door, steps in and sits down" is four actions, and the model will break at least one of them.
2. Writing the script and voiceover
The voiceover is short on purpose: seven lines in total. Two voices:
- The AI assistant (shots 1 and 4): calm and slightly synthetic.
- Mei as narrator (shots 2, 3, 6, 7, 8): warm and young.
Shot 1 | AI: Good morning, Mei. It's March 12th, 2036. Sunny today.
Shot 2 | Mei: Ten years ago, we had to remember everything ourselves. Now, my home remembers for me.
Shot 3 | Mei: The tomatoes at breakfast grow right in my kitchen.
Shot 4 | AI: Design review at nine. Your shuttle arrives in three minutes.
Shot 6 | Mei: The city got quieter. Fewer cars, more trees.
Shot 7 | Mei: Machines do a lot now. But ideas still start with people.
Shot 8 | Mei: The world ten years from now isn't colder. It just wakes up a little earlier.
Two rules made the audio much easier later:
- Narration, not dialogue. Lip-sync is still the weakest part of AI video. A voiceover over a face that isn't talking always looks fine. A face that is supposed to say a line and doesn't move looks broken.
- Each line fits inside its shot with room to spare. About 2.5 words per second, never across a cut.
3. Character reference sheets: keeping the face consistent
This is the part that matters most. AI video models are good at making a young woman in a sweater. Getting the same young woman in eight separate generations is the hard part.
I did three things.
a) A fixed character description. I wrote it once and pasted it unchanged at the start of every prompt:
Mei, a 22-year-old East Asian woman, shoulder-length straight black hair with
wispy bangs, oval face, small mole under her left eye, cream oversized knit
sweater, light gray wide-leg pants, white sneakers, thin silver smart band on
her left wrist
Pajamas for shots 1 and 2 got a second version with the same face text and only the clothes swapped. Small, specific details like the mole and the silver band help a lot. They give the model something concrete to hold on to, and they give you something to check in QC.
b) Two reference sheets. Before any shot, I generated a character sheet for each outfit and picked the best one. These were uploaded as reference images for every generation.
c) A first-frame image for every shot. This made the biggest difference. Instead of text-to-video, each shot is image-to-video: I first made a still of the exact scene (using the reference sheets), checked that the face matched, and only then animated it. The model only has to move a face that is already correct, not invent a new one.
If two shots needed to join seamlessly, I took the last frame of the first clip and used it as the first frame of the next.
4. Generating the shots
The workflow for each shot:
- Upload both reference sheets.
- Upload the shot's first-frame image.
- Paste the character description plus the motion prompt.
- Generate at 12-15 s, 720p or higher.
- Generate a few takes and keep the best one. Keep the rejected takes too, because you will want them for comparisons.
A typical motion prompt (shot 1):
[character description, pajamas] The smart-glass window slowly turns from
frosted to fully clear, warm sunrise light sweeps across the bed. She stretches,
sits up and smiles. A small glowing orb on the nightstand lights up softly.
Slow push-in, cinematic, photorealistic.
The first + last frame trick (shot 1)
My first try at shot 1 had an obvious problem: the window was supposed to go from frosted to clear, but my first-frame still already showed a clear window with full sunlight. The model can't animate a change that has already happened, so nothing changed.
The fix was to control both ends of the clip:
- First frame: an edited copy of the still with the glass frosted and the room dim.
- Last frame / target: the original sunny still.
With a frosted start and a clear end, the model only had to fill in the transition, and the window cleared properly.
General lesson: if a shot is about a change (lights on, door opens, sky brightens), the first frame has to show the "before" state. A prompt can't override a first frame that contradicts it.
5. QC: how I found the problems
Watching a clip at full speed, your brain smooths over glitches. So I reviewed every clip as frames:
# 4 frames per second, tiled into one contact sheet per clip
ffmpeg -i shot5.mp4 -vf "fps=4,scale=320:-1,tile=6x8" -frames:v 1 shot5_sheet.jpg
# or every frame of a suspicious 2-second window
ffmpeg -ss 4 -t 2 -i shot5.mp4 -vf "scale=640:-1" frames/s5_%03d.png
Then I checked a short list in every sheet:
- Does the face match the reference (eyes, bangs, the mole)?
- How many fingers are on each visible hand?
- Do props stay solid, or do they melt into hands?
- Does any body part pass through an object?
- Is any on-screen text readable and spelled correctly?
Here is everything that failed, and what fixed it.
Shot 2: the toothbrush that melted into her hand
The original idea was Mei brushing her teeth while the mirror showed her stats. In motion, the toothbrush and her fingers kept merging. The handle bent and the fingers turned into one blob.
Fix: I removed the prop. In the new first frame she stands at the sink with her hands resting on the counter, smiling at the mirror while the health widgets appear. The story is the same, and there is nothing left to glitch.
Shot 4: fused fingers, and two speakers with one voice
The first version of the assistant shot had two problems:
- Mei raises an open palm toward the orb. Open palms are where AI video most often produces extra or fused fingers, and this one had six or more.
- The model generated audio in which the assistant's line and Mei's reply were spoken by the same voice in one stream, while Mei's lips didn't move.
Fix: I made a new first frame with her hands down and out of the frame, regenerated the shot, and dropped Mei's spoken reply entirely. She just listens and smiles. The assistant's line was later replaced with a separate TTS voice (section 7), so the two characters never share a voice.
Shot 5: face morphing and walking through a car door
This was the worst one. The original prompt had Mei walk out, open the shuttle door and step in. While she walked, her face drifted noticeably from frame to frame, and when she stepped in, her leg went straight through the doorframe.
Fix: a much simpler action. New prompt: she walks a few steps toward the waiting shuttle, mostly facing the camera, slow camera push, minimal limb motion, and no entering the vehicle. The story doesn't need her to get in, because shot 6 already shows her inside.
Shot 3: a small hand glitch, trimmed instead of regenerated
The breakfast shot was good apart from a hand glitch in the last ~1.5 seconds. Regenerating would have risked losing a good take, so I just cut it: in the stitch script, shot 3 is trimmed from 15 s to 13.5 s. Cheap fixes first: before regenerating, check whether a trim, a crop or a quicker cut hides the problem.
End title: readability
A reviewer read the title as "Theifuture" instead of "The future". The text itself was correct. The problem was a thin, regular-weight font in white over a bright sunrise sky. The letters lost their edges and the space disappeared. Switching to a bold font with a dark outline and a soft shadow fixed it (code in section 9).
6. Simplifying hand-heavy actions: the rule I wish I'd started with
Looking back, nearly every regeneration came from the same thing: hands interacting with something. Brushing teeth, gesturing at a hologram, opening a door, stepping into a car.
Here is how I rewrite actions now:
| Risky | Safer |
|---|---|
| Brushes her teeth | Stands at the sink, hands resting on the counter |
| Reaches out and taps the hologram | Looks at the hologram and smiles, hands out of frame |
| Opens the car door and gets in | Walks toward the car, which is already open |
| Holds a cup and drinks | The cup is on the table and she glances at it |
| Fast walking with arm swing | A few slow steps, camera does the moving |
Let the camera and the environment carry the motion (push-ins, light changes, things moving outside a window) and keep the character mostly still. Shot 6 is a good example: Mei just sits by the window and the city does the work.
7. English voiceover with neural TTS
The generated clips came with their own audio, including the generated speech. I wanted clean, consistent English narration, so I rebuilt the audio track from scratch.
Muting the original speech
First, find where the speech is. silencedetect gives you the quiet gaps, and the speech is everything in between:
ffmpeg -i shot3_final.mp4 -af "silencedetect=noise=-35dB:d=0.4" -f null - 2>&1 | grep silence_
I muted those regions and filled them with ambience copied from a speech-free part of the same shot (birds, room tone, street hum), with short crossfades so you can't hear the patch. Shots that were nearly silent anyway just got silence.
Generating the voices
I used Edge neural TTS (the edge-tts Python package) because it sounds natural and runs from the command line:
pip install edge-tts
edge-tts --voice en-US-AvaNeural --rate=-10% \
--text "The city got quieter. Fewer cars, more trees." \
--write-media L5_mei.mp3
-
Mei:
en-US-AvaNeural, slowed 10-12%, light compression and a small amount of room echo. -
Assistant:
en-GB-SoniaNeural, slowed 5%, then processed to sound a little synthetic: band-limited to 180 Hz-9 kHz, a light chorus, a short slap echo and compression.
ffmpeg -i L4_ai.mp3 -af "highpass=f=180,lowpass=f=9000,\
chorus=0.6:0.8:40:0.25:0.3:2,aecho=0.8:0.5:35:0.18,\
acompressor=threshold=-20dB:ratio=3" L4_ai_proc.wav
Pronunciation gotcha: the TTS read "Mei" as "my". The fix was to give the TTS the spelling "May" while the subtitles and the title keep "Mei". Always listen to names.
Verifying with Whisper
To check that every line came out right, I ran the finished mix through Whisper (faster-whisper) and compared the transcript with the script. All seven lines came back intelligible and matching the script. This is a quick automated check that catches mispronunciations and clipped words you stop noticing after the tenth listen.
8. Original music, ducking and loudness
I didn't want any licensing questions, so I synthesized the music myself in Python: an additive-synthesis piano and a soft pad playing a simple I-V-vi-IV progression in C at 70 BPM, with numpy and scipy filters. No samples, no third-party recordings. It fades in at the start and swells a little under the end title.
Mixing came down to three steps:
- Sidechain ducking so the music dips whenever someone speaks (values are a starting point; tune by ear):
ffmpeg -i music.wav -i vo.wav -filter_complex \
"[1:a]asplit[sc][vo];\
[0:a][sc]sidechaincompress=threshold=0.03:ratio=6:attack=40:release=600[duck];\
[duck][vo]amix=inputs=2:normalize=0[mix]" -map "[mix]" music_vo.wav
Add the rebuilt ambience quietly underneath (about -3 dB).
Normalize loudness to about -16 LUFS integrated, which suits web playback:
ffmpeg -i mix.wav -af "loudnorm=I=-16:TP=-1.5:LRA=11" mix_final.wav
The final mix measured -15.8 LUFS, with the voice sitting clearly above the music. The audio is then swapped onto the video without re-encoding the picture, so the visuals stay byte-identical:
ffmpeg -i Mei_2036_full.mp4 -i mix_final.wav -map 0:v -map 1:a \
-c:v copy -c:a aac -b:a 192k Mei_2036_full_EN.mp4
9. Stitching it all together with ffmpeg
All eight clips go through one Bash script, so a fix to any shot is a one-command rebuild. Each clip is trimmed to its length and normalized to 1280x720 at 30 fps, then the clips are chained with 0.5 s crossfades:
D=(15 12 13.5 15 12 15 15 15) # per-shot lengths; shot 3 trimmed to 13.5 s
# normalize every input
f="$f[$i:v:0]trim=0:${D[$i]},setpts=PTS-STARTPTS,\
scale=1280:720:force_original_aspect_ratio=decrease,\
pad=1280:720:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=30,format=yuv420p[v$i];"
# chain with xfade; each offset = running total minus the overlap
cum=$(python3 -c "print($cum + ${D[$((i-1))]} - 0.5)")
f="$f[$prev][v$i]xfade=transition=fade:duration=0.5:offset=$cum[x$i];"
f="$f[$preva][a$i]acrossfade=d=0.5:c1=tri:c2=tri[y$i];"
The end title is two drawtext layers that fade in over the last ~4 seconds. This is the readability fix: a bold font plus a border and shadow.
FB="/usr/share/fonts/opentype/noto/NotoSansCJK-Bold.ttc"
STY="fontcolor=white:borderw=2:bordercolor=black@0.55:shadowcolor=black@0.5:shadowx=2:shadowy=2"
drawtext=fontfile=$FB:text='The future is already waking up.':fontsize=50:\
x=(w-text_w)/2:y=h*0.16:$STY:alpha='if(lt(t,$TS),0,min(1,(t-$TS)/0.8))'
(The film also keeps a Chinese line under it, 未æ¥ï¼Œå·²ç»é†’了, "the future has already woken up", drawn the same way 0.3 s later.)
Gotcha: xfade needs exact offsets. If one clip is shorter than the length you assume, the transition lands in the wrong place or the filter fails. Trimming every input explicitly with trim=0:${D[$i]} makes the timing deterministic.
The 1.25x final cut
At normal speed the film ran 1:49, and it felt slow. The shots are generous and the narration is sparse. For the final cut I sped up both picture and sound by 1.25x, keeping the voice pitch natural, which brings it to about 1:27:
ffmpeg -i Mei_2036_full_EN.mp4 \
-filter_complex "[0:v]setpts=PTS/1.25[v];[0:a]atempo=1.25[a]" \
-map "[v]" -map "[a]" -c:v libx264 -crf 20 -c:a aac Mei_2036_full_EN_1.25x.mp4
atempo changes tempo without changing pitch, so Mei doesn't sound like a chipmunk. If you're unsure about pacing, try this before you regenerate anything to make shots shorter.
Bonus: making the GIFs in this post
GIFs are limited to 256 colors, so I generate a palette per clip, then use ordered dithering to keep the file size down (all of these are under 5 MB):
ffmpeg -ss 24.4 -t 4 -i Mei_2036_full_EN_1.25x.mp4 -vf \
"fps=10,scale=560:-1:flags=lanczos,split[a][b];\
[a]palettegen=max_colors=176:stats_mode=diff[p];\
[b][p]paletteuse=dither=bayer:bayer_scale=4:diff_mode=rectangle" breakfast.gif
10. Lessons learned and a checklist
What I'd tell myself at the start:
- Spend most of your time on stills. A good first frame is most of a good shot. Fixing a still is cheaper than regenerating a video.
- Lock identity with redundancy: reference sheets, the same pasted description and small distinctive details, every time.
- Design around the model's weak spots. Hands, props, contact between bodies and objects, and lip-sync. Write them out of the script instead of hoping.
- The first frame must show the "before" state for any shot about a change.
- Review frames, not playback. Contact sheets at 4 fps catch what your eyes skip.
- Try the cheapest fix first: trim, then crop, then a new first frame, and only then a full regeneration.
- Own your audio. Separate voice, ambience and music tracks give you full control, and generated speech inside a clip is hard to remove later.
- Automate the build. One script from clips to final film means fixes are safe and repeatable.
- Test pacing with a speed change before you regenerate anything to make it shorter.
Pre-release checklist
- [ ] Face matches the reference sheet in every shot (eyes, bangs, mole)
- [ ] Wardrobe is consistent (sweater, pants, shoes, smart band)
- [ ] No extra or fused fingers in any frame
- [ ] No props melting, no clipping through objects
- [ ] Every voice line is transcribed correctly by Whisper
- [ ] Two speakers never share a voice. No lines without lip movement for on-screen speakers
- [ ] Voice clearly above music. Integrated loudness about -16 LUFS
- [ ] On-screen text readable on the brightest frame behind it
- [ ] The whole film rebuilds from one command
Honest limitations
- Some small things remain if you go frame by frame: background details (signs, distant people) aren't fully stable, and the robot and colleagues in shot 7 look plainly AI-generated if you look closely.
- I dropped Mei's only spoken line rather than fake lip-sync, so she never speaks on screen.
- The music is simple. It does its job, but a real composer would do better.
- The AI video model did most of the visual work. What I added was the structure around it: consistent inputs, a QC loop and a reproducible build.
If you try this workflow, I'd like to see what you make. And if you've found a reliable way to get hands right in AI video, please tell me in the comments.
Written by Hu Jin, a software and data engineer exploring AI video.
來源:DEV Community · dev.to













