Keeper 以本地 gemma3:1b 判斷重複文件並隔離其餘副本

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 3 — Slow and Steady.

Your Downloads folder has contract.pdf, contract_final.pdf, contract_final_v2(1).pdf, and contract_FINAL (copy).pdf. You know one of them is the real one. You don't know which one. So you open all four, squint at the footer, find the signed copy, close three tabs — or worse, you pick wrong and sign the wrong version.

Keeper answers that question for you. Point it at a folder: it clusters version-clone files by actual content (not filename), asks a local open-weight model which copy is the keeper and why, and quarantines the rest. Dry-run by default. It never deletes a thing.

That's the whole sloth move — a task you've been putting off for years, done by one command while you get a coffee.

Demo

_{In my case it takes time — I'm on a low-end device, and local LLMs are heavy. Sorry for the bad quality!}

A full run on my test folder: scan → spinner with a self-measured ETA → verdict table → --apply quarantines the 8 losers. Nothing was deleted.

How it works

folder ─▶ scan (hash + metadata) ─▶ extract (txt/md/csv/pdf/docx)
       ─▶ group (filename tokens + TF-IDF cosine) ─▶ decide ─▶ report
                                                       │
                                     `--apply` ─▶ _keeper_quarantine/ + manifest

Scan — walks the folder (skipping .git, node_modules, and its own quarantine), records size, mtime, sha256.
Extract — pulls text from documents; images, video and archives are recognized as media up front so their bytes are never decoded into garbage "text". A scanned PDF with no text layer is honestly flagged instead of treated as empty.
Group — byte-identical files cluster by hash; near-duplicates cluster by filename-token overlap (browser notes (1).md suffixes stripped) and content similarity. Names carrying different identity numbers (S01E01 vs S01E02, Lesson13 vs Lesson14) are never merged — episodes and numbered lessons stay apart. Pure Python, no ML dependency.
Decide — only clusters with readable text ever reach the model. Byte-identical clusters and unreadable ones (zips, images, videos) are settled instantly from metadata, so a low-end machine runs Gemma only when it actually matters. Each call goes to gemma3:1b through local Ollama (--model gemma3:4b to upgrade), temperature 0, capped output, forced JSON: {keep, reason, confidence}.
Quarantine — --apply moves losers into _keeper_quarantine/<group>/ with a manifest.json and a portable restore.sh.

The safety net

If the model's pick contradicts hard evidence — the signed copy, the newest timestamp — a transparent rule engine overrules it. Look at row 3 in the demo: the model disagreed on the contract, and the newest copy won on metadata, with the override stated plainly in the table (◐ rule — model disagreed). I'd rather ship a tool that's right than one that's confidently wrong. Rule modes are yours too: --no-model (rules only, instant) and --model-only (hard-error if Ollama is down).

The lazy payoff

Honest numbers from my own machine — a 4-core low-end box:

~/Downloads, 211 files → 35 duplicate clusters, 92 losing copies identified, in about a minute. Only 9 clusters called the model; everything else was decided instantly from metadata.
Doing that by hand means opening 92 files, comparing them, deciding. That's an afternoon I'm never getting back. Keeper: one command.
On a fast machine the model calls take seconds; on mine the spinner honestly shows minutes ([3/9] · about 40s left) — measured throughput, no fake progress bar.

Why open innovation matters here

The files Keeper reads are contracts, passports, medical letters, tax returns. Shipping filenames and text excerpts to a closed cloud model is exactly the wrong default for this job. gemma3:1b (~800 MB, runs in 4 GB RAM) reads your lease agreement on your laptop and forgets it when the process exits — no API keys, no telemetry, no network calls beyond localhost:11434.

That's what an open-weight model buys you: a private assistant with no exfiltration surface and no vendor who can change the terms next quarter. And the whole reasoning pipeline — grouping, prompting, fallback rules — is in one repo, auditable in an afternoon.

Install

curl -fsSL https://raw.githubusercontent.com/atif-1402/keeper/main/install.sh | bash

ollama pull gemma3:1b     # ~800 MB, runs on 4 GB RAM
keeper ~/Downloads        # dry run — moves nothing

Windows runs it today via pipx install git+https://github.com/atif-1402/keeper; release v0.1.0 ships standalone binaries for Linux (x64/arm64), macOS and Windows. No Ollama? Keeper falls back to its rule engine and tells you so.

Safety model

Dry-run by default — nothing moves without --apply
Never deletes — losers are moved, every byte accounted for in the manifest
Fully reversible — keeper <folder> --restore returns the folder to its exact prior state, verified byte-for-byte by tests (38 of them)