Product Autopsy 掃描作者 30 個公開 repo，18 項疑點中 4 項屬實

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

My own README told readers: “Push it before publishing this table, or remove the row until it is.”

That sentence is a note to the author. It had been sitting in a public repository for about two months, visible to everyone who opened it, and I hadn’t noticed. A tool I wrote found it.

The tool is Product Autopsy. A seller can have a polished listing and still ship a product that breaks its promises: “includes setup.xlsx”, “totals update automatically”, “ready in five minutes”. Most AI tools for sellers help write the listing. Product Autopsy checks the listing against the actual files, using deterministic checks where possible and a local model when the wording is too varied to parse. It reads a product folder the way a disappointed buyer would:

• Missing files. The docs mention a file that isn’t in the package.
• Broken links. A local link doesn’t resolve: a missing target, the wrong letter case, a path that escapes the package, or a #anchor with no matching heading.
• Wrong spreadsheet totals. A SUM range that doesn’t match its data, a total that sums the wrong column, or a result that differs from the number the README promises. A small local model proposes claims from the prose. Plain code decides what is true. A validator rejects claims that fail the source-grounding and workbook checks, leaving them for human review rather than silently repairing them.

I ran it on 30 of my own public repositories. It flagged 18 candidates. I opened every one by hand: 4 were real, all from one repository, and I fixed them in one commit. The other 14 were false alarms, and they taught me more than the hits did.

Who would pay for this

Anyone who sells a digital download and gets support emails that start with “the file you mentioned isn’t in the zip”: spreadsheet and template sellers, authors of guides with companion files, people shipping starter kits. A broken promise costs a refund, a bad review or a support thread. And the thing being audited is usually a product that hasn’t been released yet, which is why it matters that this runs on the seller’s own laptop.

My hypothesis for the business: keep the audit free and local, and charge for a written report, batch audits of a whole catalogue, or a hosted version. That is a hypothesis. Nobody has paid for this, and the “My First User” section says exactly what I did and didn’t test.

Demo

Product Autopsy is a command-line tool, so there is no deployed link. A seller runs one command on a product folder, reads a plain-language report, fixes what it lists, and runs it again before launch.

A deliberately broken sample product. The defects are planted by my benchmark script:

$ python autopsy.py samples/budget-planner --report
Product Autopsy report: budget-planner
3 problem(s) found, 6 check(s) passed

FAIL  broken link   README.md: `docs/refund-policy.md` does not resolve (missing, or the letter case differs)
FAIL  missing file  README.md: `setup.xlsx` is named in the docs but is not in the package
FAIL  spreadsheet   tracker.xlsx: `Total Revenue`: the SUM range does not match the data rows; the result differs from the number the README promises

A clean control. All 3 checks pass for this product:

$ python autopsy.py samples_phrasing/tutor-tracker-right --report
Product Autopsy report: tutor-tracker-right
0 problem(s) found, 3 check(s) passed

No problems found among the promises this version can check.

Those two are generated test products. Now a real repository of mine. Before the fixes, the scan of my 30 repos flagged 18 candidates, and I checked each one by hand. Four were confirmed defects, all in graph-density-engine. This is the tool’s output for those four rows:

graph-density-engine | missing_file | datasets/incident_01.json | experiment.md | referenced_file_absent
graph-density-engine | missing_file | incident_10.json | experiment.md | referenced_file_absent
graph-density-engine | missing_file | run_experiment2.py | README.md | referenced_file_absent
graph-density-engine | missing_file | agents/llm.py | requirements.txt | referenced_file_absent

The other 14 candidates were false alarms. My verdict on each is in verify_findings.md, and the raw scan is in real_findings_before_fix.csv.

After commit c93d44e, I rescanned. It flagged 14 candidates, the same 14 rows I had already classified as false alarms. In graph-density-engine the only remaining row is results.json, a file a command creates when it runs. The before and after scans are saved as results_before_fix.md, real_findings_before_fix.csv and real_findings_after_fix.csv.

Code

Emmimal / product-autopsy

A Python-based tool that audits product and repository claims against actual project files, identifies broken promises, and evaluates finding accuracy across deterministic, clean-phrasing, and noisy-phrasing benchmarks.

Product Autopsy

Checks whether a digital product (a spreadsheet, template pack, guide or starter kit) delivers what its listing promises. It reads a product folder the way a disappointed buyer would and tests every promise it can verify against the actual files.

Built for the DEV Hacktoberfest Open-Source AI Challenge, Week 2: Side Hustle. A small local model (Gemma 3 1B through Ollama) reads promises written in prose. Plain code decides what is true. A validator rejects model claims that fail the source-grounding and workbook checks and leaves them for human review.

What it checks

…

The repo has the checker, the benchmark and hold-out generators, the Gemma claim extractor and the scripts that produce every table in this post.

Run it

You need Python, Git and Ollama installed. Install the Python dependency and download the model once:

pip install openpyxl
ollama pull gemma3:1b

Make sure Ollama is running before starting the model experiments. Then run:

python run_all.py gemma3:1b --real

run_all.py generates the benchmark, runs the checks, clones my 30 repositories if they are not already present, scans them, runs both hold-out experiments and saves the results to results.md. Leave out --real to skip the repository scan, or leave out the model tag to skip Gemma. Once the samples exist, you can try the seller-facing checker on the deliberately broken sample:

python autopsy.py samples/budget-planner --report

The synthetic benchmarks and hold-out experiments are generated by these commands. The prompt-v1 results came from an earlier prompt version and are saved in results_1b_prompt_v1.md. The original scan of my repositories was run before the fixes; the command reuses existing repository folders and does not recreate that pre-fix state. The saved outputs include results_before_fix.md, real_findings_before_fix.csv, real_findings_after_fix.csv and my manual verdicts in verify_findings.md. The first realistic-README run, which caught 5 of 10 wrong totals, is saved in results_noisy_run1.txt.

How I Built It

The stack is deliberately small: Python’s standard library, openpyxl to read .xlsx files, and Gemma 3 1B running locally through Ollama. No API keys and no calls to outside services during a check; the model runs on the same machine.

The model reads, the code judges

The three checks themselves (missing files, broken links and spreadsheet totals) use no model. The model is there because promises come in endless wordings. A regex handles the one phrasing I wrote it for, while a model can read “revenue comes to 1,640 in total” and “the sample revenue column sums to 1,640” as the same kind of claim. The model has one job: read the prose and return claims such as “the Total Revenue cell shows 1,640”, as JSON, with the schema enforced and the temperature at 0. It never decides whether a claim is true. Every claim must pass five checks in plain code, or it goes to human review and is never repaired:

1. It matches the expected JSON structure and uses a supported claim type.
2. It comes from a file that exists in the product.
3. The claim text appears in that file (ignoring case, spacing and Markdown marks).
4. The promised number appears inside that claim text.
5. The target is one of the total labels that code read from the workbook itself, and its key word appears in the claim text. That last rule came out of a failure, described below.

Benchmark one: 30 planted defects

I generated 10 sample products with a script. Each has one planted defect per category (missing file, broken link, wrong spreadsheet total) plus valid controls, 60 valid checks in all, so false alarms can be measured. The ground truth lives in a separate file the checker never sees.

The two misses are on purpose. In two products I wrote the promised total in free phrasing (“revenue comes to 1,640 in total”) that my regex can’t parse. That gap is where a model should help.

I wrote both the generator and the checker, so these numbers show the engine works. They don’t show it finds defects in the wild. The next two sections are the ones that matter.

Benchmark two: what Gemma 1B actually did

I ran it on a CPU-only laptop with 7.4 GiB of RAM, about 1.1 GiB of it free, and no usable GPU. It took three experiments, because the first one gave an answer I didn’t trust.

Experiment 1: the main benchmark. Two of the products promise a total in free phrasing that my regex can’t parse. I tried two prompts.

Prompt v1 produced mostly noise: 23 of 30 claims were rejected. For v2, my code read the real total labels from each workbook and handed them to the model along with one example from a different domain. Rejections fell to 2, but recall did not move. All 5 accepted claims were the same sentence my regex already handled, and the model never extracted either free-phrased promise.

The two rejected claims show why the validator exists. For one product, the model quoted “Etsy Profit Sheet helps you see profit per listing. Everything lives in one workbook.” and attached the number 1537. For another it quoted “Import categories.csv into the Categories tab.” and attached 1475. Neither number appears in either sentence. Without the grounding rule, those values would have been compared against real workbook totals and could have produced false accusations against a seller.

That looked like a clear answer, but it rested on two products, and I had adjusted the prompt after seeing them. So I built a fresh test.

Experiments 2 and 3: a hold-out set. I generated 20 new products. Each has a correct workbook and one sentence promising the Total Revenue in natural wording. There are 10 different phrasings, each used twice: once with the wrong number (a defect) and once with the right number (a control). None of the sentences matches my regex, so the deterministic baseline catches 0 of 10. I did not change the prompt after seeing these results. I ran the same 20 products twice: once with a one-sentence README, and once with a realistic README around the promise (a file list, setup steps, links, and an unrelated sentence saying “every total updates automatically”).

Gemma 1B does add recall: it catches wrong totals that my deterministic checks can’t read. It is also fragile. Wrapping the same sentences in a realistic README cost 30 to 40 points of recall. In the noisy run the validator rejected 6 claims: 4 were spreadsheet claims with no number and 2 quoted text that couldn’t be found in the README. In all three hold-out runs none of the 10 correct promises was flagged, so no false accusation came out of these tests, though 10 controls per setting is a small check.

Three cautions. First, the realistic-README result isn’t stable: I ran it twice with the same code, the same data and the temperature at 0, and the results were not identical: 5 of 10, then 4 of 10 (repair-desk-wrong was caught once and missed once). The one-sentence set and the main benchmark reproduced exactly. Second, ten defects per setting is a small sample, so I wouldn’t call 80% against 40–50% a solid difference; I’d call it a warning. Third, I built both the products and the phrasings, so these results are still synthetic. They show where a small model starts to fail, not how it performs on real listings. I haven’t tested a larger Gemma, and I haven’t compared against a hosted model, so I won’t claim anything about either.

The real test: 30 of my own repos

I cloned and scanned 30 of my public repositories. The scan covered 216 checks: 206 file mentions and 10 local links. All 10 local links resolved. The file-mention check flagged 18 candidates, which I then reviewed manually.

I classified each flag as a confirmed defect, an expected absence, or uncertain, by opening the file and reading the surrounding text.

The four real rows come from three causes, all in graph-density-engine:

• A leftover note to the author in the README, “Push it before publishing this table”, about a table row that no longer existed.
• Missing data files. During the original scan, the README's Quick Start loaded incident_01 through DatasetLoader, but the required dataset JSON files were absent. On a fresh clone at that time, the Quick Start failed with FileNotFoundError. I reproduced it before restoring the files. I regenerated the datasets with fixed seeds, but I cannot prove they are byte-identical to the originals used for the published experiment.
• A stale comment in requirements.txt pointing to agents/llm.py, a file that doesn’t exist, while the README says there is no model anywhere in the pipeline. The fix is one commit: c93d44e. I couldn’t find my original data files, so I regenerated them with the generator’s fixed seeds. The generator uses a fixed seed for each scenario, so the files should match, but I can’t prove they are byte-identical to the ones behind my published numbers, and I’d rather say so than imply it.

After the fix, a rescan flags 14, and every one is a false alarm I had already classified. Zero confirmed defects remain, which shows the find-and-fix loop works. It doesn’t show the repos are clean. The scan outputs from before and after the fix are saved in the repo, so the 18-to-14 change can be checked.

Precision on real repositories was 4 of 18, about 22%, and all four came from a single repo. I’d call the file-mention check useful for finding candidates and not yet good enough to trust unattended. Before scanning my own repos I tuned the noise filters on four third-party template repos, so those four don’t count as a clean test. The only change I made after seeing my own results was the path fix described below.

Two bugs the checker had

Running on real machines found two bugs in my own tool.

• Windows hid case-mismatch links. On my Windows laptop, broken-link recall was 80% and overall recall was 87%, lower than on Linux. My code used Path.resolve(), which on Windows rewrites a path to its real on-disk case, so Guide.MD pointing at guide.md looked fine. It would break for a Linux reader. I replaced it with plain path arithmetic, and the numbers matched on both systems.
• Paths that climb a folder. A mention of ../RESOLVER_RULES.md was flagged as missing even though the file exists. I fixed the path handling. This is a correctness fix made after seeing the real-repo results, and I’m reporting it as one.

What it can’t see

• It skips fenced code blocks to avoid noise from examples, so a file listed in a README’s code-fenced tree can slip through. My graph-density-engine README lists phase2_results.json in its project tree, and that file isn’t in the repo. The checker missed it.
• It reads files and never runs a product’s instructions. A Quick Start that fails when you run it is invisible to a tool that only reads. Running the instructions, the “buyer disappointment simulator” in my original pitch, is the obvious next build.
• I have no recall number for real repos. I know what it flagged. I don’t know what it missed.

What I’d build next

1. Test a larger Gemma on the realistic-README hold-out set, where the 1B model drops to 40–50%.
2. Flag a file mention only when it sits in an instruction like “open”, “copy” or “run”, to cut the false alarms, and test that on a fresh set of repos.
3. Run the Quick Start itself.

Why Does Open Innovation Matter?

• The product folder never leaves the machine. The thing being audited is often unreleased work. Sending a seller’s unpublished files to a hosted API is a poor trade for a pre-launch check, and with a local model it isn’t needed.
• The model is replaceable. The checker doesn’t care which model fills the claim slot. Swapping Gemma 1B for a larger local model means changing one tag, and I can measure the difference with the same benchmark.
• No per-audit API fee. The audit runs locally with Python, openpyxl and a downloaded Gemma model. There is no hosted-model charge per check, although the model still needs storage, hardware and electricity. That is what makes a free local tier possible at all.
• The design doesn’t depend on trusting the model. The validator is designed to reject unsupported claims before they become findings. That reduces one class of model error, but it doesn’t guarantee that every accepted finding is correct. In my runs, a 1B model caught up to 80% of the wrong totals my regex couldn’t read, fell to between 40% and 50% with realistic text around the promise, and produced no false alarms on the controls, though the tests are small. Where open fell short: the 1B model is fragile. Recall dropped from 80% to between 40% and 50% when I added realistic text around the promise, and a larger local model might fix that, which I haven’t tested. I haven’t compared against a hosted model, so I can’t tell you whether a closed API would have done better. What I can say is that an open model made the privacy and cost story possible, and the validator provided a checkable boundary around what the model could submit as a finding.

My First User

My first and only user so far is me, on 30 repos. No seller has tried it, and nobody has paid. I won’t claim that anyone would, because I don’t know.

What I do know is that it surfaced four things I had missed in my own published work, and that the false alarms are as informative as the hits: they show exactly where a seller would get annoyed. The next test that matters is handing it to a seller and asking whether the flags helped or irritated them.

Prize Categories

Best Use of Gemma

Gemma 3 1B, run locally through Ollama on a CPU-only laptop, reads README prose and returns the promises it finds as schema-constrained JSON. On a hold-out of 20 products with promises my regex can’t read, it caught 8 of 10 wrong totals in short READMEs and 4 or 5 of 10 in realistic ones, where the deterministic baseline caught none. A validator rejects anything it can’t ground in the source. The tables are in “How I Built It”.