BubbleFacts 如何以來源核查約束 3B 模型生成直播歌曲資訊
Keeping a 3B model honest: source-checked song facts for live streams
BubbleFacts 以 Wikipedia 文章為依據,讓本機運行的 Llama 3.2 3B 生成直播歌曲資訊,並在顯示前逐條進行程式核查。年份、專輯和作曲者等基本資料直接取自 Wikidata 和 MusicBrainz;Wikipedia 內容則採用經 72 小時沉澱的較早版本。不受來源支持的內容會被捨棄;串流者可在歌曲播放後標記錯誤,記錄只留在本機、不會上傳。
BubbleFacts is a free, open-source desktop app for musicians and DJs who stream. While someone plays a song, it shows short facts about that song as bubbles in OBS. The facts are written by a small language model running on the streamer's own computer.
That last part is the hard part. A 3B model will happily tell your viewers that a song won a Grammy it never won. On a live stream there's no one to catch it: the musician's hands are on the instrument. So the design question was never "how do we get good facts?" It was "how do we make sure a wrong one never reaches the screen?"
Here's what we ended up with.
1. Give the model nothing to invent from
The pipeline runs in a fixed order:
song facts → Wikipedia → AI → screen → Wikidata / MusicBrainz → custom facts
Basic facts (year, album, composer) don't need a model at all. They come straight from Wikidata and MusicBrainz.
For the rest, the app first finds the song's Wikipedia article (grounding), and the model writes only from that text. A few things matter more than you'd expect:
- Which article. Search results are full of near misses: a newer game with the same name, an artist page, a disambiguation page. A lot of the code is about rejecting the wrong page, not finding a right one.
- Which part of the article. The lead comes first, since Wikipedia writes it to stand alone. Then come the sections that report checkable things (composition, release, charts). Reception sections are filtered, because "critics called it a masterpiece" turns into an unattributed claim in a 3B model's hands.
- Which version. Recent edits are the least reviewed. The app uses the settled older version of an article (we call it the 72-hour rule).
2. Screen every caption against the source
Grounding lowers the error rate, but it doesn't reach zero. Small models merge clauses, drop hedges, move a year to the wrong event, and swap who did what to whom. So every caption goes through screening before it's shown, and anything the source doesn't support is dropped.
The checks are deliberately boring, deterministic code, not a second model:
- Names: every person, role, award winner and platform in the caption must appear in the source.
- Support: find the source sentence the caption retells, then compare numbers, connectives and relations.
- Attribution: who said or did it, and whether a day or year went missing.
- Frames: unnamed references, a sentence's limits ("one of", "in the UK"), and chart details.
- Wording: dropped hedges, misplaced years, wrong owners, reversed roles.
- Repair: some scope words a 3B model drops can be put back from the source instead of throwing the fact away.
Each check lives in its own small file, so a fix to one rule reads one file. The entry point is screenClaims in screening/screen.ts.
3. When in doubt, show nothing
The cost of a missed fact is a quiet moment on stream. The cost of a wrong fact is a musician being corrected in their own chat. So the app abstains automatically: no review screen, no "approve this fact" step, because a musician mid-song can't click anything.
When a wrong fact does slip through, the streamer can press Wrong after the song. That logs the song, the caption, the article and the sentence it came from (locally, never uploaded), so the case can be replayed against the screening code later.
4. Local by default
The built-in model is Llama 3.2 3B, run through node-llama-cpp inside an Electron app. Nothing goes to a cloud AI. One surprise: without a system prompt of our own, node-llama-cpp gave the model its own "helpful assistant" prompt, which changed the output compared with Ollama. If you switch runtimes, check what prompt you're actually sending.
5. Governance, built in on purpose
Accuracy checks are one layer. We also wanted the whole project to be something a streamer could trust without reading the code, so we built a set of deliberate controls around it.
What can reach the screen
-
A content standard in code.
content-standard.tsholds word lists for things no article sentence, caption or data fact may say. Streams are watched by all ages, so the filter runs on the source text before the model sees it, and on the output after. - Settled sources only. The 72-hour rule above keeps fresh, unreviewed Wikipedia edits out.
- Abstain over guess, as above: a missing fact is always the safe outcome.
Privacy and local control
- The server answers only to this computer. It checks the Host header to block DNS rebinding, and the desktop app accepts requests only from its own window.
- Secrets stay out of logs. Sign-in tokens are never logged; the log says only "still signed in" or "expired".
- The miss log is private. The Wrong log stays on the streamer's machine, is included in their backups, is removed with their data, and is never attached to a bug report.
Supply chain and releases
- Protected main. Every change goes through a pull request and required checks; no force-pushes or deletes.
- Locked tags, immutable releases. A published release and its version tag can't be changed. A bad release gets a new version, not a quiet swap.
-
Verifiable downloads. Each release ships
SHA256SUMS.txt, a build attestation you can check withgh attestation verify, and an SPDX SBOM. - Signed updates. The in-app updater checks a release signature against a key built into the app, and test builds are never offered to users.
AI accountability
- Disclosure everywhere. The README says plainly that most of the code was written with Claude Code, with a human reviewing and testing every change, including on live streams.
-
Traceable commits. AI-assisted commits carry an
Assisted-by:trailer, and contributors are asked to do the same. - Independent review. Each release gets an outside review, and changes during release candidates get a second review before merge.
None of this is exotic. The point is that it was decided up front, written down in the repo, and enforced by GitHub settings and tests rather than by memory.
What we'd tell someone building something similar
- Ground first, generate second, and treat the source as the only allowed input.
- Write the checker as plain code you can unit test, not another model you have to trust.
- Make abstention the default, and design for users who can't review anything.
- Keep a local log of every miss, and replay it after each change.
- Decide your governance controls early, write them in the repo, and let settings and tests enforce them.
The app, including the screening code, is MIT-licensed: https://github.com/frolicchris/bubblefacts
BubbleFacts was built with AI assistance (Claude Code), and so was this write-up.
來源:DEV Community · dev.to