ShopSnap：將 WhatsApp 訂單截圖轉為每日銷售報告，AI 全程在本機運行

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

ShopSnap turns WhatsApp order screenshots into a daily sales report. It is made for small shops that take orders over chat.

Lots of small sellers run their whole business inside WhatsApp. A customer writes "2 kg rice and 1 dozen eggs" and the owner replies with a price. By evening the orders are spread over dozens of chats. Who ordered what? Who paid? Who still owes money? Most owners answer that by scrolling back through every chat or by trusting their memory.

Here is how ShopSnap handles it:

1. Drop in the screenshots and pick the business date.
2. A local vision model reads each chat. A local text model turns it into orders.
3. You review and fix every order. Nothing is saved until you click Confirm.
4. You see total sales and who has not paid and a total for each customer.
5. Mark orders as paid (full or part) and copy a ready follow-up message to paste into WhatsApp.

Chats that show only a saved name go to a Pending list. You attach a phone number once and ShopSnap remembers that name from then on.

Who would pay for it: any shop owner who sells over chat and keeps the books by hand. The app is free and MIT licensed today. The version I would sell is a packaged desktop app with the setup done for the shop owner. That part is future work.

Demo

Upload and review. Drop screenshots then check and fix every order before saving.

Pending. Chats that show only a name wait here until you attach a phone number.

Daily report. Total sales and who has not paid and a total for each customer.

All screenshots use fake demo data.

Code

sasmeee / shopsnap

Turn WhatsApp order screenshots into a daily sales report. Runs fully locally with Ollama. Customer data never leaves your PC.

ShopSnap

Turn WhatsApp order screenshots into a daily sales report. Built for small shops.
The AI runs on your own PC. Customer numbers and sales never leave it

Built for the Hacktoberfest Open-Source AI Challenge, Week 2: Side Hustle.
The repository was started on October 15, 2026, inside the challenge window.

• Demo video: Watch on YouTube

• DEV post: ShopSnap: WhatsApp order screenshots to a daily sales report, with AI that stays on your PC

Screenshots

Upload and review. Drop screenshots, then check and fix every order before saving.

Pending. Chats that show only a name wait here until you attach a phone number.

Daily report. Total sales, who has not paid, and per-customer totals.

All screenshots use fake demo data.

What it does

1. Drop WhatsApp screenshots and pick the business date.
2. A local vision model reads the chat. A local text model turns it into orders.
3. You review and…

How I Built It

Stack: Next.js (App Router and TypeScript) with shadcn/ui and Tailwind. SQLite through better-sqlite3. Ollama runs the models. zod checks the JSON. libphonenumber-js handles phone numbers and sharp resizes the images.

Open models, both run locally through Ollama:

• gemma3:4b reads the screenshot and writes out the chat as text.
• qwen2.5:3b takes that text and extracts the orders as JSON.

The pipeline:

screenshot -> resize -> vision model (text) -> text model (JSON)
-> code checks -> you review -> SQLite -> report

Two small models on a normal PC will make mistakes. So I did not build the app around trusting them. I built it around checking them. The model proposes and plain code decides:

• A phone number is the only customer ID. A chat name never creates a customer.
• Code decides whether the chat header is a number or a name. The model does not.
• Every amount the model returns must appear in the chat text. Items must add up to the total. A failed check blocks the Confirm button.
• "Paid" is accepted only if the chat has payment words. Otherwise the status becomes unknown and the owner decides.
• The same screenshot twice is never counted twice. I hash it and store the order position.
• Only the hash is stored. The image is never kept.

I tested the pipeline on 10 fake screenshots with bun run eval:

One screenshot did most of the damage. Chat 07 holds two orders. The model returned one and misread its amounts. That single chat caused 15 of the 17 wrong fields. The other miss was in chat 09 where the model misspelled a name and guessed "unpaid" when the status should have been "unknown".

Two wrong orders got past the code checks. One of them is the order the model never returned. Nothing can validate an order that is not there.

What I learned: a 4B model is weakest on chats with more than one order. The checks are good at catching wrong numbers but they cannot catch a missing order. A forced re-read can find an order the first read missed. The review screen is the last safety net before anything is saved.

Real chats are messier than my fake ones so expect lower numbers. That is why the review screen is not optional.

Why Does Open Innovation Matter?

A shop's chats hold customer phone numbers and daily sales. That is private data. With a closed API every screenshot would leave the shop's PC and travel to someone else's server. Many owners would never accept that and I would not blame them.

Open weights changed three things for ShopSnap:

• Privacy: the models run on the owner's own machine. Numbers and sales never leave it.
• Cost: there is no bill for each screenshot. A shop that processes 40 chats a day pays nothing extra.
• Control: the models are two environment variables. If a better small model comes out, you change VISION_MODEL or TEXT_MODEL and you are done. It also works offline after the first download.

The honest trade is accuracy. Small local models are weaker than the biggest hosted ones. I answered that with code checks and a human review step instead of a bigger model.

Both models keep their own licenses. Gemma 3 is under the Gemma Terms of Use and Qwen2.5 3B is under the Qwen Research License. The README tells anyone to read both before using ShopSnap in a paid setting.

Prize Categories

• Best Use of Gemma: Gemma 3 (gemma3:4b) is the vision model that reads every screenshot. Without it nothing else in the pipeline has any text to work on.