跳到正文
DEV Community· zaqero AI·· 7 天前AI 評分45

Zaqero AI 如何在 AI 影片中保留產品標籤真實度:無需綠幕的追蹤器

Keeping a product's label real in AI video: a tracker with no green screen

AI 導讀

Zaqero AI 建立並測試一套無需綠幕的產品標籤追蹤及印字替換流程,以單一參考影格追蹤標籤,避免逐格追蹤累積誤差。流程在一段蠟燭罐測試影片中錄得 0.04-0.05 px RMS 抖動;標籤處理尚未投入生產,產品照片不經重繪及改用 24 fps 渲染則已上線。

正文

If you make ads for a small brand with an AI video model, the first complaint is rarely the motion or the light. It is the label. A jar comes back with letters that are almost right, a logo that has melted a little, a line of small print that is now a different line of small print. To the brand that is not their product any more.

We build Zaqero AI, which makes short product videos for small businesses. This post is about how we try to keep the label theirs: what is in production, what we built and measured on a test shot, and what is still open.

1. Do not let anything redraw the photo first

The cheapest fix is upstream. An earlier version of our pipeline "cleaned up" a product photo with an image model before handing it to the video model, setting the product on a plain grey ground. That meant the label the video model saw was already the image model's copy of it.

Now the product photo reaches the video model as it is. We cut the product out of its background and set it on light grey with a margin, keeping every product pixel; when the background is not plain enough to cut, the photo goes in as it is, only turned upright. Nothing is regenerated or restyled on the way. This part is live.

It helps, and it does not solve the problem: the video model still generates every frame, and small print drifts.

2. Find the label in the generated shot

After a take comes back, we ask a vision model one narrow question about one frame: where are the four corners of the label (and, for a round thing like a jar, the left and right edges of the jar at the label's height)? It answers in JSON, as shares of the frame.

Code then checks the answer before anything trusts it: four points, inside the picture, a real four-sided shape, and big enough, at least about 1.5% of the frame. If the label is not clearly there, the answer is "not found" and the shot stays exactly as it was generated. That rule runs through everything below: when unsure, change nothing.

3. Follow it without a green patch

The obvious way to follow a label is to key it by colour. On our test jar that failed on the label's own lettering: the key jumped up to 28 px between frames.

What works is following the picture itself. Our patch tracker takes the label region from one reference frame and, on every other frame, searches for the move and scale (and, for things that turn, the rotation) that best match it, by normalised correlation, refining in smaller and smaller steps down to a tenth of a pixel.

Two details mattered more than the search:

  • Every frame is compared with the reference, never with the frame before. The previous frame only says where to start looking. Chained frame-to-frame matching drifts; matching against one reference does not accumulate error.
  • Follow the label, not a box around it. Our first version followed a box that included some of the room behind the jar, and the top and bottom of the label trembled against each other. Following the label's own corners on every frame fixed it.

On the test shot, a slow push-in on a candle jar, the first version matched the reference at 0.993 on average. The final version, measured on the rendered video against the picture, sits at 0.04-0.05 px RMS of shake, 0.14 px at worst. A track that matches worse than 0.85 at its worst moment is thrown away and the shot left alone.

4. Change the print, not the label

Our first attempts pasted a new label image over the old one. It looked like a sticker every time: too sharp, too clean, frozen while the real label underneath kept catching the light.

So we stopped painting labels. On each frame we flatten the label region out of the picture, fit a smooth curve through it to get the surface's own light (dark side and bright side), and treat what stands apart from that curve as print. Only the lettering's pixels are filled in, from that same frame's clean surface around them and with its own fine noise. The label's edges, its bow, its light and its grain stay the footage's. Then the new print, if the brand asked for one, goes on in the same place, at the same size and shape. (That is also our rule: a label is edited in place, as it is, never covered by a different one.)

To keep the print from looking pasted on, we measure the shot instead of judging it by eye:

  • Softness: the blur of the shot's sharpest edges in pixels. The print is blurred to match. On the test shot the footage measured 1.2 px and the print came out at 1.4 px (our first, unmeasured attempt was 0.65 px: far too sharp).
  • Flicker: how much a bare part of the label changes from frame to frame. A version that added fresh grain each frame on top of footage that already had its own noise flickered at 0.97 levels a frame against the original's 0.50. With the print lifted from each frame's own pixels it measured 0.54.

5. A bug we found on the way: 24 into 30

While measuring the shake we found that every video had a stutter we had stopped noticing. The video model returns 24 frames a second; we rendered at 30, so every fifth frame was a repeat. A label tracked to a twentieth of a pixel still looks rough if the footage under it hitches. We now render at 24. This change is live.

6. What is not solved

Plainly:

  • The label path (steps 2-4) is built and tested, not switched on in production yet. Everything in sections 3 and 4 was measured on one test shot, with the vision step stubbed, on our own rendering bridge. No paid generation has been through it end to end.
  • The flattening is bilinear between four corners, which is close enough for a label seen nearly square-on and not for a steep angle.
  • Reflections on glass are imitated, not measured from the real lights.
  • Hands over the label, labels that leave the frame and labels that fold are not handled; the shot stays as generated.

We will write again when it has run on real videos, with those numbers.

If you are working on the same problem, the two ideas we would keep are: match against one reference, never the previous frame; and lift the old print off each frame instead of laying anything over it.

Zaqero AI makes short product videos for small businesses from their own photos: https://www.zaqero.app

來源:DEV Community · dev.to