Amar Khobor 將 56 條 WhatsApp 訊息整理成 48 秒語音簡報

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥

You slept. It listened. Here's what you missed.

Last night, my father sent four voice notes to our family WhatsApp group. He was planning a trip to Shantiniketan, setting a budget, and — most importantly — updating us on my grandfather's health after a scare that initially sounded like a heart problem.

I was asleep.

By morning, I would have had to play all four notes, remember who said what, and mentally assemble a to-do list. Instead, I played a 48-second audio briefing that caught me up on all of it — in a warm voice, with a receipt for every claim.

This is the story of Amar Khobor (আমার খবর) — a briefing tool born the night a speech-to-text model mistook "heart problem" for "samosa at the market."

The night it mattered

Here is what Whisper-large-v3 produced from my father's voice note about Dadu:

আমি হেবে ছিলাম হাটে সমসা কিন্তু দাদু ভললো গ্যাসে সমসা

And here is what Sarvam's Saaras produced from the exact same audio:

আমি ভেবেছিলাম হার্টের সমস্যা, কিন্তু দাদু বলল গ্যাসের সমস্যা।

Whisper wrote "trouble at the market" (হাটে সমসা). My father said "heart problem" (হার্টের সমস্যা). One is a snack. The other is a family medical emergency.

That single mistranscription is why Amar Khobor exists. Family WhatsApp groups don't sleep. Overnight, voice notes pile up in Bengali, Hindi, English, Benglish, and Hinglish — and catching up means playing every note, remembering who said what, and mentally assembling a to-do list. Every morning. Every family member. Nobody has time for that.

What I built

Amar Khobor turns a WhatsApp group export into a warm, sub-minute audio briefing plus structured, verifiable show notes.

• ▶ Listen to the sample briefing — generated from my family's real overnight voice notes, narrated with ElevenLabs.
• ▶ Try the live app
• Read the code

The digest arrives as two things at once:

• A spoken script — warm and conversational, like a family member catching you up. No timestamps, no citations read aloud.
• Show notes — structured cards below the player, mapping every claim to its sender and timestamp: Baba → Dadu's gastric issue, not heart → health → urgent.

Here's what it produced from my family's actual messages that night:

"Shobai kemon acho? Amar Khobor is here with a little update from Baba. Baba is already dreaming about our next trip! He's suggesting Shantiniketan—staying in a nice lodge and visiting the Sonajhuri Haat. He also mentioned a budget of around 2000, though he says we might spend a bit more if we enjoy some shopping there. So do think about it, okay?
On a more serious note, there was a little scare with Dadu. Baba thought it might be a heart issue, but Dadu mentioned it's just gas. Edkat went to check on him, and Baba is also heading there. Thankfully, Dadu is doing much better now and has already had some khichdi. Baba will be taking him to see Dr. Sen tomorrow morning. So please, don't tension too much and everyone get some good sleep. We'll catch up in the morning. Shuvo ratri!"

The sloth math

Fifty-six messages. Four voice notes. Forty-five seconds of audio you cannot skim.

Morning-me would replay all four notes to find the budget and the health update, then scroll the chat to check who said what. The briefing is 48 seconds, and the receipts are already attached.

The saving isn't the seconds — it's that audio has no Ctrl-F.

Sloths don't catch up faster. They let the night do the work.

The bake-off: what survived a human check

I didn't want to pick a transcription provider based on someone else's benchmark table. So I built a bake-off harness that scores Saaras and Whisper-large-v3 on my own family's audio.

One honest caveat first: the WER ground truth in test_data/manifest.yaml is the Saaras transcript (with one speaker name corrected from the chat log). So Saaras's 0.00% WER is an upper bound, not an independent accuracy score — Saaras agrees with itself by construction. The evidence that survived a human check is the diff below.

The clip that matters — bn_003, verified by ear:

Full per-clip results, including the noise-guard test: test_data/eval_results.json · demo/stt_comparison.md

That Whisper hallucination is real: on a one-second clip of near-silence, it produced the word "you" from nothing. Saaras produced nothing — correctly.

Why this matters for the product: Saaras is purpose-built for Indic languages and code-mixed speech, and it's what runs in production. Whisper is the open-weights baseline I benchmark against — inspectable, reproducible, and swappable.

The bug I shipped and fixed

Gemma 4 did something I didn't expect: it occasionally emitted its entire chain-of-thought — planning, drafting, self-correcting — before the JSON digest. Sometimes it wrapped the JSON in an array instead of returning a bare object.

My first extractor took the first { to the last }. That grabbed a { from mid-reasoning and produced invalid JSON. The pipeline crashed on a response that contained a perfectly good digest — just buried in chatter.

The fix was a tolerant backward extractor: find the last occurrence of "spoken_script", backtrack to the { that opens that object, and slice to the closing }. Now the parse succeeds even when Gemma rambles — and the prompt is hardened to discourage the rambling in the first place.

idx = content.rfind('"spoken_script"')
start = content.rfind('{', 0, idx)
end = content.rfind('}')
cleaned = content[start:end+1]

This turned a flaky LLM response into a deterministic parse — committed, with tests, in pipeline/summarize.py.

The engineering decisions worth explaining

Why filter audio before STT? Voice notes are the expensive part. An energy-based VAD pass tags silence and background noise before any API call, so a forwarded chain of muted clips costs nothing. Forwarded messages are also excluded from summarization — nobody needs a briefing about a chain-forwarded good-morning graphic.

Why one LLM call for both outputs? The spoken script and the show notes describe the same events. Generating them in a single structured Gemma call keeps them consistent — if the script says Dadu needs a doctor, the show-notes card for it exists too.

Why a 202-and-poll architecture? A real overnight export takes a minute to process. POST /digests/run returns 202 Accepted immediately, FastAPI BackgroundTasks does the work, and the frontend polls every 2 seconds. No Celery, no Redis, no message broker — this is a family tool, not a distributed system.

Why ephemeral memory by default? Family voice notes are some of the most private audio that exists. Audio and briefings live strictly in-memory during the session and vanish on server restart. MongoDB Atlas is an opt-in adapter, never a requirement.

My first users were my family

I didn't recruit beta testers — I used my own family's real WhatsApp group. Baba sent real voice notes about a real trip and a real health scare. The pipeline transcribed them with Saaras, Gemma summarized them, and ElevenLabs narrated the briefing in a warm Bengali-speaking voice.

The show-notes card for the health update read: Baba → Dadu's gastric issue, not heart → health → urgent. That's the receipt. Not a vague summary — a claim, tied to a sender, tied to a timestamp, flagged as needing attention.

Why open models matter here

This is a tool for family voice notes — some of the most private audio that exists. That constraint shaped every choice:

• Gemma 4 summarizes the digest because it's an open-weight model I can swap, self-host, or point at any provider without rewriting the pipeline. The provider is switched with one environment variable.
• Whisper-large-v3 via Groq is the open-weights baseline, so the STT comparison is between two models whose behavior I can inspect — not a black box I have to trust.
• Edge-TTS keeps development free and local-first; ElevenLabs is the production voice for the live deployment, opt-in and never a hard dependency.
• Ephemeral in-memory storage means zero infrastructure setup and zero database footprint by default.

The honest asymmetry: Saaras — the model that wins my bake-off — is closed. I can't inspect or self-host it. I picked it for Bengali accuracy and kept the provider swap one environment variable away, which is the only honest position for a model you don't own.

Honest limitations

• Gemma reasoning & schema wrapping — Gemma occasionally emitted chain-of-thought before the JSON and sometimes wrapped output in an array. The prompt is hardened and the extractor is tolerant; it's now deterministic.
• WER ground truth is Saaras-derived — the 0.00% is an upper bound, not an independent accuracy score. The bn_003 diff above is the human-verified evidence.
• STT accuracy on heavy code-mixing varies — Saaras handled my Benglish clips well; Whisper struggled. Four clips is an anecdote, not a benchmark — the harness exists for exactly this reason.
• The VAD is energy-based, not neural — cheap and fast, but a loud TV can fool it. Silero is the obvious next upgrade.
• Render Blueprint targets Starter — the committed render.yaml provisions an always-on Starter instance; the free tier would add a cold start after 15 minutes idle.
• No authentication on the API — single-family deployment assumption. Multi-user needs auth and rate limits.
• Benglish ground truth is manual — there's no standard Romanized-Bengali orthography, so bake-off ground truths are typed by hand.

Prize categories

• Best Use of Gemma — Gemma 4 powers the single-pass dual-output digest (spoken script + show notes), with a hardened prompt and a tolerant backward JSON extractor that turns flaky LLM output into a deterministic parse.
• Best Use of ElevenLabs — ElevenLabs narrates the production briefing in a warm Bengali-speaking voice; it's the voice in the committed sample MP3 and the live deployment.
• Best Use of Render — the backend deploys as a containerized FastAPI service via a committed Dockerfile and render.yaml Blueprint, with a root health route and Vercel-aware CORS.

Sarvam's Saaras isn't a prize track, but it's central to the open-models story: it's the production STT that made the "samosa vs heart problem" comparison possible in the first place.

Try it

• ▶ Listen to the sample briefing
• ▶ Try the live app
• Read the code

Built for the Hacktoberfest Open-Source AI Challenge, Week 3: Slow and Steady. Speech-to-text by Sarvam AI and Groq; briefing voice by ElevenLabs.