Sloth Scribe：拒絕無法證明主張的文稿編譯器

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

I planted a lie in a README: "Cuts build time by 60% compared to the old pipeline." There is no benchmark anywhere in that repo. I ran Sloth Scribe on it three times. Gemma wrote that claim into its draft all three times, and all three times my verifier refused to let it through, marking it NEEDS INPUT with the reason "metric needs measured evidence."

That refusal is the whole project.

What I Built

Sloth Scribe is a local write-up compiler. It reads a git repo into a fact ledger, and a local Gemma model drafts claims that cite ledger fact ids. Plain code then decides which claims are supported. The model can phrase things, but it cannot create facts or issue verdicts.

It saves the tedious part of a write-up: reconstructing what you actually built from git history. Maintainers and hackathon builders who keep postponing their release notes and project posts get to be a little lazier. They get a draft in seconds, and every sentence is labeled VERIFIED, INTERPRETATION or NEEDS INPUT, so reviewing it takes less trust and less rereading.

Git-to-write-up tools already exist, so I am not claiming novelty there. My narrower claim is that unsupported claims are refused instead of guessed, and you can see why each one was refused.

Demo

Recorded run (start here): https://sloth-scribe.vercel.app/replay
Home page: https://sloth-scribe.vercel.app/

The replay is a recorded run, not live generation. The Run button on the home page needs a local Ollama install, so it only works on your own machine.

On the replay page, click the sentence about the 60% build-time cut. The evidence panel shows its status and the reason it was blocked. Click a verified sentence, such as the commit count, to see the fact ids and evidence locations behind it (a commit hash or a file with line range).

Code

mohanprasath-dev / sloth-scribe

Local write-up compiler. Reads a git repo into a fact ledger, a local Gemma model phrases claims by ID, and a code verifier blocks anything unsupported as NEEDS INPUT. Runs offline.

Sloth Scribe

Local evidence-constrained write-up compiler. Reads a git repo into a deterministic fact ledger, a local Gemma model phrases claims citing fact IDs, and pure code verifier blocks anything unsupported as NEEDS INPUT. Runs 100% offline.

Started Oct 23, 2026 (Week 3 window Oct 19-25) for the DEV Hacktoberfest Open-Source AI Challenge, Week 3: Slow and Steady.

What It Is

Sloth Scribe is a local tool that turns git history into reliable release notes and write-ups without hallucinated facts.

Scans a local git repository (commits, diffstat, README, package.json).
Builds a deterministic fact ledger where every fact has an ID, value, source, and exact evidence location (commit hash, file and line range).
Prompts a local Gemma model via Ollama to draft sentences as structured claims citing fact IDs.
Pure deterministic TypeScript verifies every claim against the ledger and labels each sentence: VERIFIED, INTERPRETATION, or NEEDS INPUT.
Unsupported claims are refused…

Started Oct 23, 2026, inside the Week 3 window.

How I Built It

Open-source AI used: gemma3:4b-it-q4_K_M as the primary model and gemma3:1b as a fallback, both open-weight Gemma models. They run locally through Ollama, using its structured JSON output so the model has to return claims in a fixed schema. The app is Next.js and TypeScript.

The pipeline has five stages: scan, ledger, generate, verify, export.

Ledger. Deterministic code reads commits, diffstats, files, dependencies, scripts and README lines. Every fact gets an id and an evidence location, for example git:log, commit:cc4f592 or file:package.json:10-10.
Generate. Gemma sees only a compact ledger, never raw diffs. Each claim must cite fact ids.
Verify. Pure code, no model call. Five rules:
- R1: numbers and dates must match a cited fact.
- R2: counts come from git, not README text.
- R3: named features and packages must exist in dependencies, files, scripts or commit subjects. The README saying "dark mode" does not count.
- R4: metrics such as percentages need a measurement file. README assertions never count.
- R5: no citation, no claim.

Interpretations are allowed only if they contain no numbers, dates, paths or identifiers.

Export. Verified and interpretation sentences come out as text. Every NEEDS INPUT sentence becomes a visible [NEEDS INPUT: reason] placeholder.

For example, "There were 14 commits made" is verified against git:log. "The project supports CSV export" is verified by commit cc4f592 and src/export.js.

The test repo is a synthetic fixture with 14 commits and two seeded falsehoods: the 60% metric and a "dark mode" feature that no commit implements.

I built it in phases using an orchestrator that ran a builder agent and a separate verifier agent per phase. The verifier re-ran every phase's checks before anything merged to main, because I do not trust an agent's own "done."

What went wrong first

My first full run verified only 3 of 12 claims. Two true dependency claims were refused because my version comparison choked on the ^ in ^12.0.0. A true CSV-export claim was refused because Gemma cited a README line instead of the code. These were my bugs, not the model's. I fixed both, and three runs afterward gave 6, 7 and 6 verified claims out of 12.

Across those three runs:

No false claim was marked verified.
The 60% claim appeared in all three runs and was refused each time.
The dark mode claim appeared in one of the three runs and was refused. In the other two, Gemma never wrote it.
CSV export was verified in all three runs.

Honest limits

The verifier covers numbers, dates, counts, and named features, dependencies and scripts. Adjectives and causality can only be interpretation.
It is conservative. Some true claims are still refused, and a few refusals carry a reason that does not match the rule that actually fired. For example, a correct repo-name claim got a misleading reason.
A claim naming two dependencies is checked only against the first, so compound claims can be verified on partial evidence.
gemma3:1b was too small for this task. In my swap check it produced one claim and none verified. I make no claim that verdicts match across models. The verifier's logic does not depend on the model, and my handwritten test drafts show that, but I did not measure parity with the 1B model.

Why Does Open Innovation Matter?

A closed API with a privacy setting can write fine prose, so "open means private" is not my argument. My argument is about where the trust boundary lives.

The boundary is code you can read. The model is replaceable. The verifier is plain, unit-tested TypeScript that decides what reaches the page, and anyone can audit it or tighten a rule.
Inference stays local by default. The default path calls only Ollama on localhost, with no per-request cost after the model download. The repo also has an optional hosted mode through the Gemini API. It is off unless a key is set, and when it is on, it sends the compact ledger to Google, so my local claim applies to the default path.
Open weights make this testable. I could swap model sizes, pin exact tags, and measure the real cost on a laptop GPU. That is how I learned the 1B model could not do the job.

The Lazy Payoff

Measured on an RTX 4050 laptop GPU (6 GB), gemma3:4b-it-q4_K_M:

Generation took 10.5 to 11.7 s in the three runs above, and 14.7 to 20.9 s in earlier runs. I did not isolate why it varied.
Peak GPU memory was 5066 MiB, a whole-GPU nvidia-smi reading on a card with 6141 MiB total, so it fits with little headroom.

I did not time a manual write-up, so I make no minutes-saved claim.

Prize Categories

Best Use of Gemma: a local Gemma model through Ollama is the generation layer, and this post shows what it can and cannot do.