以 Whisper 與 VLM 標註流程微調 YOLO26 交易直播畫面分類器

Most computer-vision demos start with a dataset someone else curated. Mine started with screen recordings I was not allowed to redistribute - which turned out to be the most instructive constraint in the whole project.

This is the story of trade-state-classifier: a fine-tuned YOLO26n-cls model that looks at a screenshot from a trading livestream and answers one binary question - is there a filled position visible on screen right now? - plus the labeling pipeline that made it possible without anybody's data leaving my machine.

Why binary classification (and not object detection)

The upstream goal is an event pipeline: watch a livestream, detect the moment a streamer's position state changes (flat -> open -> flat), extract the displayed entry/stop/target levels at that moment, and hand a validated event to the execution layer.

Full UI-element detection (order tickets, entry lines, price badges) is the endgame - but it needs far more labeled variety than I could curate responsibly. A state gate needs to answer exactly one question reliably. So v1 = one classifier, one binary decision, measured hard.

The labeling problem

Fine-tuning means labels, and labels for this task have a nasty property: open position is rare. Uniform frame sampling at 1 fps gives you thousands of near-identical flat-screen frames and a handful of meaningful ones. Label quality would drown.

The flywheel that solved it:

1. Whisper (faster-whisper, local) transcribes the VOD. Talking traders narrate what they are doing.
2. Speech-cue parser finds candidate windows - phrases around "getting filled", "taking profit", "stopped out", "flat now".
3. Sparse frame extraction - only the seconds around each cue, not the whole hour. Roughly two orders of magnitude fewer frames to look at.
4. Local VLM (qwen2.5vl:7b via Ollama) drafts labels for those keyframes, based on an element list (position badge, PnL overlay, position box).
5. Confluence confirmation - a window only survives if speech and vision agree. The two chatterboxes must contradict each other to flag a frame for me.
6. Owner-verified dense segments around confirmed trade windows - I confirm every frame in the segments I train on.

That last step matters. VLMs fast-draft; they do not carry truth. Every training frame was verified by the only model with skin in the game: me.

The training run

• Base: YOLO26n-cls (Ultralytics), the nano variant, deliberately - CPU inference at ~15ms/frame keeps the state gate cheap enough to run continuously.
• Input: ~1,700 confluence-confirmed crops, 512x512.
• 20 epochs, batch 16.
• Evaluation: temporal validation - sessions the model never saw during training. Adjacent frames leak; a random frame-split lies to you. Holding out entire sessions is the honest split.

Result table (iterations v11 -> v17):

The interesting run: a session whose only state was "no trades, all day" produced ten 1-second VLM "open" blips taught nothing - until dense windows around the one confirmed trade window were added. Sparse VLM labels had a blind spot; the fix was density, not cleverness.

The publishing decision that took longer than the training

After validation the question was: what can go public?

The model can. The frames cannot - they are someone else's screen, and I decided redistribution crosses a line I did not want to argue with anyone about.

So: weights on the Hugging Face hub, zero images anywhere, no channel names, no video IDs. Then a full audit of my own public repos - because the training story lives in commit history, docstrings, and example paths. Every identifier got generalized (channel-a/channel-b, session placeholders), the harness scripts removed those, and history was rewritten rather than patched so nothing was merely invisible-but-recoverable.

A "publish without exposure" checklist that I now treat as standard:

• Weights are fine, frames are not.
• Descriptive metrics OK; provenance narratives are leaks.
• git log, example paths, and docstrings are part of your public surface - audit them, not just README.md.
• If exposure retrofitting needs three layers (working tree, history, hub), treat that cost as the price of skipping hygiene in the first place.

What the classifier still cannot do

Saying this publicly protects future-me:

1. It detects a visible position badge - a streamer chart-hopping mid-trade looks like flat.
2. It is platform-appearance-specific. A position box on one platform does not look like another's; pooled training helps, new platforms will need new data.
3. A drawn (hypothetical) position-tool box and a filled position can look identical. The visual gate alone cannot tell you a fill happened - that is why the upstream pipeline requires speech confluence and state-transition logic.

What is next

• ONNX export and an in-browser demo: the whole model running in the visitor's browser via onnxruntime-web, on screenshots the visitor brings (privacy-compatible demo by construction).
• More platform calibrations, pooled rather than forked.
• The universal detector: entry/stop/target/quantity reads off any chart, with OCR and geometric validation deciding whether an event is even possible.

The full labeled pipeline, validation tooling and inference scripts are in the stream-signal-vision repository, and the published weights are at trade-state-classifier on Hugging Face.