Clipjack 如何在 4 分鐘內從 6 小時直播中找出爆紅片段
How we find viral moments in a 6-hour stream in 4 minutes
Clipjack 團隊分享其流程,目標是在約 4 分鐘內處理 6 小時 VOD,找出適合剪成 Shorts 的片段。流程將直播切成每段 15 分鐘、重疊 30 秒的片段,並行處理四段,先利用音訊、聊天室和 160p、2 fps 影片訊號篩選候選時間窗。昂貴模型只處理候選片段,判斷保留與否及精確起訖點;最後只渲染 10 至 20 個入選片段。
A GTA VI streamer this week sits in front of a camera for 6 to 10 hours. Somewhere in there are maybe ten moments worth a Short. Watching it back to find them takes longer than the stream. We built Clipjack to do that part, and the target was a 6-hour VOD in about 4 minutes of wall time. This is how the pipeline is shaped.
1. Cut the stream into pieces and run them side by side
The slow way is to run a model over every frame. A 6-hour 1080p60 VOD is about 1.3 million frames. Instead the ingest step cuts the stream into 15-minute pieces with a 30-second overlap (so a moment on a boundary isn't cut in half) and processes them four at a time. Each piece is read once and produces cheap signals, and the output is one per-second signal table for the whole stream.
2. Audio and chat carry most of the signal
For streams, the streamer's voice is the best detector there is. Each audio piece goes through speech-to-text with word timestamps, which gives us loudness, pace, laughter and the words themselves ("no way", sudden silence after a crash). On Twitch and Kick we also replay chat: messages per second, emote spikes (LUL, KEKW, OMEGALUL), clip-command spikes. When chat and the streamer spike at the same second, that's almost always a moment.
3. Video signals are cheap, on purpose
Video runs on a 160p, 2 fps proxy, not the source. From it we read hard cuts, motion, flashes, the WASTED/BUSTED grey-out, HUD changes, stillness, and where the facecam box is. None of this needs a GPU. It confirms or rejects audio and chat candidates, and it tells us what to trim (menus, map screens, loading).
4. A judge, not a threshold
Every second gets a fused score, and the peaks become candidate windows. Only those go to the expensive model: 6 to 12 frames, the transcript window and the context around it. It decides keep or drop and, more importantly, the exact in and out points. That part matters most. When we measured 180 GTA Shorts from channels that get reach, 82% had the conflict on screen by 0.5 s and the payoff at a median 86% of the clip, with the cut 0 to 2 s after. Most raw clips fail on trimming, not picking.
5. Render last, and only what survives
Only the 10 to 20 picks get rendered: vertical 9:16, facecam detected and stacked over the gameplay, word-level captions. Rendering is the most expensive step per second of output, so it runs on the smallest set.
The numbers
Parallel pieces plus cheap signals is what turns hours into minutes. The expensive model only ever sees the candidates, a small slice of the stream. If you're building anything that has to search long video, the lesson we'd pass on: let cheap signals cut the haystack, and spend the model on boundaries, not on detection.
Demo: https://youtu.be/UoLW5tiUFLA. We launch on Product Hunt on Tuesday. Happy to answer questions on any stage.
來源:DEV Community · dev.to