LastWhisper 將本機錄音轉錄並提供語義搜尋

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

I built LastWhisper, a local-first voice memory archive that helps people preserve, search, and revisit the stories hidden inside their recordings.

The idea behind it is simple: a voice recording can outlive the moment, place, or even the person who created it. That felt like a natural interpretation of Ghost in the Machine. The “ghost” is not an AI pretending to be someone. It is the real voice, story, and memory that already exists inside the recording. The machine simply helps you find it again.

With LastWhisper, a user can:

Import WAV, MP3, M4A, or FLAC recordings into a private local archive.
Transcribe audio locally with open Whisper inference.
Read timestamped transcript segments and click one to seek the original audio.
Search naturally, even without remembering the exact words used in the recording.
Inspect genuine transcript evidence instead of receiving an invented answer.
Open a search result at its exact source timestamp.
Save important evidence as a memory and add a personal title or note.
Rebuild the semantic index without requiring a separate vector database.
Optionally generate transcript-grounded summaries with a locally configured Ollama model.

The default setup does not require a hosted transcription API, embedding API, cloud LLM, analytics service, or remote storage.

Code

LastWhisper

LastWhisper is a local-first voice memory archive. It imports personal audio, transcribes it with open-source Whisper inference, indexes transcript evidence with local sentence-transformer embeddings, and lets people search, replay, and save source-grounded memories.

No hosted transcription, embedding, LLM, analytics, or telemetry service is required. Model files may need a one-time download before offline use.

Features

Private local audio archive for WAV, MP3, M4A, and FLAC files.
CPU-first transcription with faster-whisper, language detection, progress, retry, and cancellation.
Native playback and timestamped transcript segments that seek into the source audio.
Optional source-grounded summaries through a user-configured local Ollama model.
Local semantic search with normalized sentence-transformer embeddings stored in SQLite.
Evidence snippets with recording IDs, segment IDs, dates, and source timestamps.
Saved memories with immutable source evidence plus editable titles and notes.
Rebuildable, transcript-revision-aware search index.
Additive migration support for existing Phase 1, Phase 2, and early Phase 3 databases.
Responsive…

The repository includes:

Native CPU installation instructions.
Docker Compose with persistent archive and model-cache volumes.
Architecture and privacy diagrams.
Model and third-party license documentation.
CI, dependency audits, migration coverage, and offline unit tests.
A separate opt-in integration test for real cached AI models.
Contribution, security, and code-of-conduct documentation for public open-source development.

How I Built It

The frontend uses React, TypeScript, Vite, Tailwind CSS, and Lucide. It communicates with a loopback FastAPI backend. SQLAlchemy and SQLite persist recordings, transcript segments, semantic chunks, saved memories, and embedding metadata. Original audio is stored outside the web root under generated UUID filenames.

Local transcription

LastWhisper uses faster-whisper with CTranslate2. The verified default path uses the pinned SYSTRAN faster-whisper-base conversion on CPU with int8 compute.

The worker stores:

Detected language.
Ordered transcript text.
Start and end timestamps for every source segment.
Processing progress and safe retry/cancellation state.

Clicking a transcript segment sets the audio player to that segment's real Whisper timestamp.

Local semantic search

For retrieval, LastWhisper uses sentence-transformers with the Apache-2.0 paraphrase-multilingual-MiniLM-L12-v2 model pinned to an immutable revision.

Long transcripts are split into bounded overlapping chunks. Every chunk remains linked to its recording and contributing transcript segment IDs. The generated 384-dimensional vectors are normalized and stored directly in SQLite as little-endian float32 BLOBs alongside:

Model and revision metadata.
Vector dimension.
Transcript fingerprint.
Source segment IDs.
Faithful source text.
Start and end timestamps.

Search embeds the query locally, computes cosine similarity, applies a configurable threshold, and returns ranked transcript evidence. The score is presented as a ranking signal—not a probability or guarantee of correctness.

Source-grounded memories

Saving a result creates an immutable snapshot of the source evidence while keeping a foreign-key link to a real transcript segment. Users can edit only the memory title and personal note; editing never rewrites the transcript.

Deleting a recording cascades through its transcript segments, semantic index, and saved memories. Re-transcription requires explicit confirmation because replacing source segment IDs can invalidate memories tied to the old transcript.

Verification

I tested the release on a Ryzen 5 5600H with 14 GiB RAM:

Real Whisper base CPU/int8 inference detected English and produced timestamped transcript segments.
Real sentence-transformer inference produced normalized 384-dimensional embeddings.
Semantic search ranked the intended recording across multiple transcript memories.
Both model caches reloaded with offline flags enabled and outbound socket creation blocked.
The real-model integration test passed upload, transcription, indexing, search, source attribution, memory editing, deletion cleanup, and restart persistence.
38 offline backend tests passed.
Two frontend DOM tests passed, including source timestamp navigation.
TypeScript, ESLint, production build, Docker health checks, npm audit, pip-audit, and git diff --check passed.

The full verification record is here:

https://github.com/Th-Shivam/Lastwhisper/blob/master/docs/VERIFICATION.md

Why Does Open Innovation Matter?

Voice recordings can contain identity, health information, relationships, locations, family history, and moments that were never meant to become somebody else's training or analytics data.

A closed transcription or embedding API would make this archive depend on another company's:

Retention and privacy policy.
Pricing and rate limits.
Network availability.
Model behavior.
Product lifetime.

Open models made a different architecture possible.

After the model files are cached, LastWhisper can transcribe and search without uploading the recording or transcript to a hosted AI provider. The user can inspect where every result came from, replace a model, rebuild the index, back up ordinary files and SQLite, or continue using the archive even if an external service disappears.

Open also made the system more honest. The application records the exact model revision and embedding dimension that produced an index. If that configuration changes, the index can be rebuilt instead of quietly comparing incompatible vectors.

At the same time, I did not want to misuse the word “offline.” Initial dependency and model downloads require internet access. LastWhisper exposes cache state, supports local-files-only modes, and fails with actionable errors when required weights are unavailable instead of pretending inference succeeded.

My Agent Session

I built and release-hardened LastWhisper with meaningful AI-agent assistance through OpenCode. The process included repository audits, migration and concurrency reviews, real-model debugging, privacy hardening, frontend accessibility work, Docker verification, and adversarial release review.

I have not embedded the raw private session transcript because it contains local development paths and operational details that require curation before public sharing. The public engineering trail is preserved in 31 focused commits, automated tests, the verification report, and the repository's architecture/security documentation.

Commit history:

https://github.com/Th-Shivam/Lastwhisper/commits/master

The Spirit of It

The most important moment in LastWhisper is not when a similarity score appears.

It is when someone clicks Open source, the recording jumps to the exact timestamp, and the original voice plays again.

The model does not replace that voice. It does not impersonate it or generate a polished version of what it thinks was said. It helps the listener return to the real evidence—the pause, tone, words, and context that made the memory meaningful in the first place.

That is the spirit of this project: using a machine not to manufacture a ghost, but to keep a real voice from being lost inside a forgotten audio file.

Prize Categories

I am entering LastWhisper for the overall Hacktoberfest Open-Source AI Challenge: Week 4 prize.

I am not claiming a partner prize category because the core application intentionally uses local open-source components rather than adding a sponsor technology only for eligibility.

Team submission: Solo project.

This submission and implementation were created with meaningful AI assistance. I reviewed the technical claims, code, verification results, and final article, and will set DEV's AI disclosure to some_ai before publishing.