Transcribe Video to Text 即時轉錄説話者標籤錯置的排查與修正

I build Transcribe Video to Text, a Chrome extension that turns any video playing in a tab into text while it plays: YouTube, course platforms, webinars, a local file opened in Chrome. For interviews and calls it also labels who is speaking.

Last week I recorded a short demo interview for a YouTube video about the extension. Two synthetic voices: Maya, the host, asks questions, and Daniel, the guest, answers. I exported the transcript to Word and the host's questions sat under the guest's label. Not once, again and again.

It turned out to be two separate bugs, one on the server and one in the side panel, plus a limit of live diarization that I could not fix. Here is the whole path, from capturing tab audio in Manifest V3 to the labels.

How tab audio gets to the speech model in MV3

There is no background page in MV3, and the service worker can't hold a media stream. So the capture is split across three places.

1. The service worker gets a stream ID. When the user clicks Transcribe in the side panel, the worker calls chrome.tabCapture.getMediaStreamId({ targetTabId }). This only works for a tab where the user actually invoked the extension. If Chrome refuses, the panel doesn't show the raw error. It says: "To transcribe this tab, click the Transcribe icon in the toolbar above. This grants audio access for the current tab."

2. An offscreen document turns the ID into audio. The worker creates it with chrome.offscreen.createDocument({ reasons: ['USER_MEDIA'], ... }) and passes the stream ID. The offscreen page calls getUserMedia with chromeMediaSource: 'tab' and that ID.

One thing you notice in the first minute of testing: capturing a tab mutes it for the user. The fix is to route the captured stream back to the speakers:

audioCtx = new AudioContext();
audioCtx.createMediaStreamSource(mediaStream).connect(audioCtx.destination);

3. MediaRecorder sends small chunks over a WebSocket. Opus in WebM at 32 kbps, a chunk every 250 ms. The recorder starts only after my server answers ready, which means the connection to the speech model is already open, so the first words have somewhere to go.

A small trap here: the first chunk is the WebM header, about 220 bytes, and it arrives even when the tab is completely silent. I used "first chunk arrived" to switch a "sound is coming in" indicator on, and it lit up on a video with no sound at all. In that session the server got 233 bytes in 37 seconds. Now the first chunk is ignored, and the indicator waits for a later chunk that actually carries audio.

4. The server relays audio to Deepgram. A live connection with nova-3, interim_results, smart_format and diarize: true. Deepgram sends back messages with the text, an is_final flag, a speech_final flag at the end of an utterance, and a list of words. With diarization on, every word carries a speaker number.

Bug 1: one label for the whole message

Note where the speaker lives: on each word, not on the message. My server took the speaker of the first word that had one and put that label on the whole message:

const speaker = alt.words?.find(w => w.speaker != null)?.speaker ?? null;

Most of the time a final message is one person, so this looked fine. But when the host asks a short question and the guest answers right away, one final message can hold both voices. The whole thing went out under the label of whoever spoke first.

The fix splits every final message into runs of consecutive words by the same speaker. A word without a speaker number joins the current run:

function splitBySpeaker(words) {
  const runs = [];
  for (const w of words || []) {
    const cur = runs[runs.length - 1];
    const curSpk = cur && cur.find(x => x.speaker != null)?.speaker;
    if (cur && (w.speaker == null || curSpk == null || w.speaker === curSpk)) cur.push(w);
    else runs.push([w]);
  }
  return runs;
}

If there is more than one run, each run goes to the client as its own final segment with its own start time, duration and speaker. Interim messages are left alone: they get replaced a moment later anyway.

The test has five cases: question and answer in one message, a single speaker, an interim message, a word without a speaker number, and a response with no diarization at all. The two-voice case fails on the old code, all five pass on the new one. The server fix went live on October 18, so transcripts saved in the web account got correct labels right away.

Bug 2: the side panel closed the paragraph one chunk too late

The panel doesn't print every final chunk as a separate line. It collects chunks in a buffer and closes a paragraph when the text ends a sentence and is longer than 80 characters, or grows past 250 characters, or has 4 chunks, or the speaker changes. The paragraph gets the label of its first chunk.

The bug was in the order of operations. A new chunk was pushed into the buffer first, and only then did the code check whether the speaker had changed. So on a speaker change, the guest's first sentence was already inside the host's buffer and went out as part of the host's paragraph, under the host's label. The Word export copies the panel, so the document had the same mistake.

The fix is boring and that's the point: if the speaker changed, close the previous speaker's paragraph first, then push the new chunk.

speakerChanged && deepgramChunkBuffer.length && flushBuffer();
deepgramChunkBuffer.push({ text, start: start || 0, speaker: speaker ?? null });
// ...then the usual length and sentence checks

I reproduced the demo interview in a browser test: load the side panel page, push the same sequence of transcript messages into its listener, read the paragraphs from the DOM. On the old code two checks failed, exactly the way my Word file looked. On the new code they pass. This part shipped in version 1.3.5 on October 21.

A bonus bug found while checking the fix

While I was checking labels on a TED interview, I found that the server only stored messages that had speech_final. A long reply can produce several is_final messages before the utterance ends, and everything except the last one was lost in the saved transcript. In one recording, the stretch from 0:29 to 0:54 was saved as a single word. The live panel showed it fine, so it went unnoticed since July.

Now the server keeps is_final text in a buffer and writes it out on speech_final, on a speaker change, or when the connection closes.

The limit I could not fix: live diarization is not stable

The same TED interview had another complaint: the whole first minute was labeled as one speaker, and the second label showed up at 1:01 in the middle of a sentence. This time my code was not the problem. Deepgram itself sent speaker 0 for everything up to 60.6 seconds.

I ran a small experiment with the same SDK, nova-3, diarization on, English, on an 80 second clip from that talk:

• Live stream, run 1: speaker changes at 12.8, 27.3, 37.0, 57.9 and 78.0 seconds. Correct.
• Live stream, run 2, same file, same parameters: speaker 0 from 12.8 all the way to 78.0. Wrong, exactly what the user saw.
• The whole file sent as a prerecorded request: correct, and it even caught a short reply at 17.3 seconds that both live runs missed.

So in live mode, with two voices that sound alike, the labels can be perfect on one run and collapse into one speaker on the next. The fix I see is to keep the session audio in server memory and, after the user presses Stop, send it once more as a whole file with diarization and relabel the saved segments. That roughly doubles the speech model minutes per session, so it is not shipped yet.

What I took away

• With diarization, the speaker belongs to the word, not to the message. Anything that labels a whole message by its first word will mislabel quick back-and-forth.
• Buffering for readability has an order of operations. Close on the boundary before you add what is past it.
• A synthetic two-voice interview is a cheap and repeatable test for speaker labels. It caught both bugs in one recording.
• Before you blame your code for wrong labels, log what the model actually sent. One of my two "label bug" reports was the live model, not my code.

If you want to try it on your own interviews or lectures, the extension is here: Transcribe Video to Text. YouTube videos with subtitles are free and unlimited, AI transcription of other videos has a free monthly allowance.