nightshelf 用 4B 視覺模型為 321 段素材影片建立語意搜尋索引

This is a submission for the Hacktoberfest Open-Source AI Challenge, Week 3: Slow and Steady.

What I Built

I make short videos, and all of them draw from one shared folder of stock clips: 321 files from Pixabay and Pexels, 7.6 GB, about two and a half hours of footage. Every file is named with a number. pixabay/4926.mp4 could be anything.

There is a catalog next to the files, with tags, but the tags are the search queries I typed when I downloaded each clip. They tell me what I was looking for that day, not what the clip contains. So when I need "a single candle on a dark background", the honest method is to open the folder and scrub through thumbnails. I have done that more times than I want to admit.

nightshelf is a small command-line tool that looks at the library once, unattended, and writes down what is in every clip. For each video it pulls three evenly spaced frames, asks Gemma 3 (the 4B vision model, running locally through Ollama) to describe them, embeds the description, and stores everything in a SQLite file. After that I just type what I want:

$ nightshelf search "a candle flame" --limit 3
1. pixabay/4926   score 0.0328  sim 0.83  10.6s
   A single candle burns brightly against a dark background, casting a warm, flickering light…
   status: prayers-nightwatch=approved | used: prayers-nightwatch
2. pixabay/4933   score 0.0320  sim 0.80  33.1s
   A single candle with a flickering flame burns brightly against a dark background…
3. pixabay/51749  score 0.0320  sim 0.82  20.0s
   A single lit candle burns brightly in a dark room, casting a warm glow against a black background…

The tool also reads my existing catalog (read-only, it never writes to it), so it knows which clips I already approved, rejected, or used for a given project. --status unused means "something I have not used yet and did not reject".

The real run on my library: 321 clips described and embedded, 0 failures, 2,874 seconds in total, 8.9 seconds per clip, on my Mac, with no GPU server involved.

Demo

The video (about 80 seconds, a real terminal on my real library) shows the overnight run with its one-line-per-clip progress, the stats of the full run, a few searches, and an export of a shortlist.

These are real terminal outputs from my library (the path of my clip folder is masked):

Code

Deval123 / nightshelf

Overnight, local video-clip librarian: an open-weight vision model describes your stock clips so you can search them by meaning.

nightshelf

Overnight indexer that makes a local video clip library searchable by meaning.

A stock-clip library grows until nobody remembers what is in it, and file names like clip_0412.mp4 say nothing. nightshelf looks at a few frames of every clip with an open-weight vision model running on your own machine, writes a short description, stores it in a local SQLite database, and lets you search the library from the command line ("rainy city at night", "someone walking a dog on a beach"). Nothing leaves your network, the library is only ever read and the whole library is processed unattended, overnight.

How it works

flowchart LR
    L[("Clip library<br/>read-only")] --> S["scan + sync<br/>(new / changed / removed)"]
    C["catalog.json<br/>read-only"] -. "filters, status, uses" .-> Q
    S --> F["ffmpeg<br/>N frames per clip"]
    F --> V["Vision model<br/>(Ollama)"]
    V --> D[("SQLite index<br/>descriptions + FTS5")]
    D --> E["Embedding model<br/>(Ollama)"]
    E --> D
    D --> Q["search / export<br/>full-text

…

Apache 2.0, Java 21, no framework, 169 tests. The README has a quick start (jar or Docker), every command, and a configuration table.

How I Built It

The pipeline is deliberately boring:

clip library (read-only) → scan + sync → ffmpeg (3 frames) → Gemma 3 via Ollama
      → SQLite (descriptions + FTS5) → embeddings → search / export

One structured answer per clip. I ask Gemma for JSON matching a schema: a description of at most two sentences, a list of subjects, a setting, a time of day (day, night, dusk, dawn, unknown) and a mood. Temperature is 0, and 0.3 on the retry after a malformed answer, so a retry does not repeat the same bad output. Everything the model returns is treated as untrusted: the JSON is validated, lengths are capped, control and bidirectional characters are stripped, and the prompt tells the model that any text visible in a frame is content to describe, never an instruction.

Hybrid search without a vector database. 321 vectors do not need a server. Search runs SQLite FTS5 for the words and a plain cosine similarity in Java over the embeddings (nomic-embed-text, also through Ollama), then merges the two rankings with reciprocal rank fusion (k = 60). If Ollama is not reachable at search time, it falls back to full-text only and says so in one line.

SQLite, on purpose. One machine, one writer, a few hundred rows. The database uses WAL, foreign keys and versioned migrations, every statement is parameterized, and it refuses to open on a network filesystem, because SQLite locking on SMB or NFS is not something I want to trust. The clip library is mounted read-only, in Docker as well.

Slow and steady, as a design rule. The indexer handles one clip at a time and commits each result in its own transaction. A SIGTERM finishes the current clip, prints a summary and exits cleanly. A test simulates a hard kill and checks that the next run redoes only the unfinished clips. Ten failures in a row stop the run, because Ollama is probably down. And if the library drive is unmounted, the scan sees zero clips, so a guard refuses to flag more than half of the index as removed. An unplugged disk should never look like a deletion.

Things that did not work the first time. Almost every description began with "The video shows…". A sentence in the prompt reduced it, a display-time cleanup removed the rest, and the stored text and embeddings are untouched. The query "spring flowers in daylight" matched nearly everything because of the word "in", so stop words are now removed from the full-text query. User text is always split into words and quoted before it reaches FTS5, so operators like NEAR or * are plain text.

Why Does Open Innovation Matter?

My library is not secret, but it is mine: 7.6 GB, plus a record of every choice I made about it. A hosted vision API would mean sending frames from every clip to someone else's servers, and paying again every time I add a clip.

With open weights, none of that applies:

• Nothing leaves the machine. Frames and descriptions go to a model on localhost. The README says plainly that traffic to Ollama is unencrypted http, so it must stay on localhost, a trusted LAN, or an SSH tunnel.
• No per-image cost, no quota. Indexing 321 clips cost electricity and about 48 minutes of waiting.
• It works offline once two models are pulled.
• The model is pinned. The index was built with gemma3:4b. Nobody can change it under me, and swapping models is two environment variables (VISION_MODEL, EMBED_MODEL).

There is a real trade-off: a larger hosted model would probably describe some clips better. For this job, a 4B model that I can run unattended on my own computer was enough, and I would rather have an index I fully control.

The Lazy Payoff

I waited once, for 48 minutes, while doing something else. Now finding a clip takes a second instead of a scroll through hundreds of thumbnails, and export turns a query into a shortlist file (Markdown, CSV or JSON) that I can keep next to a project. The tool never copies or moves a clip.

It has limits, and they are worth stating. Concrete visual queries work well: "a candle flame" returns candles, with similarity between 0.80 and 0.83 for the top three. Abstract ones do not: "hope after a storm" returned wind turbines, raindrops on a window and a rainy night street, with similarity between 0.58 and 0.63, and I would not trust it. Three frames per clip can also miss fast action. For my use, the approach is good enough, and I know where it stops.

Prize Categories

Best Use of Gemma. Gemma 3 (4B, vision) runs locally through Ollama and produces the description of each of the 321 clips: structured output with a JSON schema, a retry strategy, and sanitized results. It is not fine-tuned.