Fridge Sloth：以 Gemma 3 離線辨識雪櫃食材並生成購物清單和健康餐點

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Fridge Sloth is for the moment you stand in front of the open fridge, door wide open, trying to remember what you need to buy and what on earth you could cook tonight.

Instead, you take one photo. Fridge Sloth:

1. Looks at the photo and lists the food it can actually see, with a rough "plenty / some / low" amount.
2. Checks it against your staples, the things you always want at home (milk, eggs, tomatoes…), and builds a grocery list of what's missing or running low.
3. Suggests three healthy meals you can make mostly from what's already there: real dishes like omelettes, salads, stir-fries and wraps, 30 minutes or less, 5 steps max. Anything a recipe still needs goes onto your grocery list with one tap.

Who gets to be lazier? Anyone who does the weekly shop and then stares at a half-full fridge wondering what's for dinner. You skip the mental inventory, the list-writing and the "what can I even make with this?" moment.

Everything runs locally on Gemma 3 (4B) through Ollama, on my 6-year-old M1 MacBook. No account, no API key, and your fridge photo never leaves your machine.

Demo

The video is a real run of the app on my laptop with a public-domain USDA fridge photo (source). The two waits are sped up and labelled with their real time: ~37 s for the scan and ~26 s for the recipes. I recorded it before the speed-up described below; the same scan now takes ~20 s. The voice-over was generated with ElevenLabs.

• Gemma saw: milk, yogurt, juice, eggs, lettuce, cheddar cheese, bread, pesto…
• Grocery list: butter, tomatoes, carrots, apples, orange juice, plus spinach and feta, which I'd just added to my staples. None of them are in the photo ✅
• Healthy meals: a pineapple & lettuce omelette (20 min), a cheesy lettuce wrap (15 min) and a quick yogurt bowl (10 min). One tap put the omelette's olive oil on the list.

Code

finallyjay / fridge-sloth

🦥 Snap your fridge, get a grocery list and healthy meals — runs fully offline on Gemma 3 + Ollama.

Fridge Sloth 🦥

Snap a photo of your fridge, get a grocery list and three healthy meals you can cook with what's already there Runs fully offline on Gemma 3 through Ollama. Your fridge photo never leaves your machine.

Built for the Hacktoberfest Open-Source AI Challenge, Week 3: Slow and Steady.

How it works

┌──────────────┐   1. vision (no hints)   ┌──────────────────────┐
│ fridge photo │ ───────────────────────> │ items actually seen  │
└──────────────┘        Gemma 3           │ + amount left        │
                                          └──────────┬───────────┘
┌──────────────┐   2. match by name                  │
│ your staples │ ────────────────────────────────────┤
└──────────────┘     plain code, ~1 ms               │
                                          ┌──────────▼───────────┐
                                          │ grocery list         │
                                          │ (missing / low)      │
                                          └──────────┬───────────┘
                     3. healthy recipes              │
                          Gemma 3         ┌──────────▼───────────┐
                                          │ 3 meals, ≤ 30 min    │
                                          └──────────────────────┘

1. Vision: Gemma lists the food it can actually see in the photo, with a rough amount (plenty, some, low) It is not…

It's deliberately tiny: a Python server that uses only the standard library (http.server + urllib), an optional tracing module, and one HTML page with vanilla JS. The only thing you install is Ollama:

brew install ollama && ollama pull gemma3:4b
python3 server.py   # open http://localhost:8000

There are 35 tests, and Gemma is mocked in them, so they run in CI without a model.

How I Built It

Stack: Gemma 3 4B (vision) on Ollama, a zero-dependency Python server, and a single HTML page. The page shrinks phone photos to 1024 px in the browser before sending them, which is plenty for Gemma and much faster.

Structured output everywhere. Every call to Gemma passes a JSON schema in Ollama's format field, so the server always gets back data it can use ({"seen": [{"item": "eggs", "amount": "plenty"}]}), never prose to parse.

The bug that taught me the most: Gemma "saw" whatever I told it to look for

My first version sent the photo together with the staples list and asked "which of these are in the fridge?". On the USDA photo, Gemma confidently reported butter, tomatoes and apples as "low". None of them are in the picture. Handing a small vision model a checklist primes it to find the items on it.

The fix was to split the work:

1. Vision step, no hints: "List every food you can actually see. Never guess hidden items." The staples list is not in this prompt at all.
2. Matching step: your staples are compared with the items Gemma actually listed.

That one change removed the invented items. At first, step 2 was a second, text-only Gemma call. It's now plain code, and the Sentry traces below are why.

The same rule, never trust the model's claim, check it, came back for the recipes. In one run a recipe listed a banana "from your fridge" when there was none. Now any recipe ingredient that wasn't actually seen is moved to the grocery list.

Asking a small model for less

• I first asked Gemma to also classify each item (ingredient, drink, condiment…). The 4B model was bad at it ("eggs: condiment"), and those wrong labels starved the recipe step of ingredients. Removing the field worked much better.
• "Healthy" needs to be spelled out. The first recipes included a "smoothie & cheese toast". The prompt now says what a healthy meal is (a real dish built on vegetables, whole grains, legumes, eggs, fish or lean protein; easy on butter, cream and salty sauces; never a drink presented as a meal) and that each meal must use at least two foods actually in the fridge.
• Matching staples doesn't need a model at all (see below).

Watching the agent with Sentry Agent Tracing

A scan took about a minute, and I wanted to know where that minute went. Fridge Sloth has optional Sentry Agent Tracing, off unless you set SENTRY_DSN:

• Each scan or recipe request is an invoke_agent trace with a gen_ai.chat span per Gemma call, carrying the model, token counts and Ollama's own timings (model load, prompt processing, generation).
• Every check the server runs on Gemma's output is a gen_ai.execute_tool span: dedupe_seen_items, match_staples_by_name and check_recipe_ingredients. In Sentry's Agent Activity view, the "never trust the model" checks show up right next to the model calls, with what they caught.

1. The traces exposed a real bug. On a messy example photo, two scans failed after about 4½ minutes. The trace showed why: a single vision call ran for 4.46 minutes, and the next step never started. The 4B model was stuck in a loop, listing the same items until its JSON answer was cut off at about 10,000 characters.

The fix: the schema now allows at most 30 items, which Ollama enforces while generating; every answer is capped in length; and a cut-off answer fails fast with a clear message. The same photo then scanned in 1.15 minutes, and that trace showed the next problem.

2. The traces showed where to optimize. The matching step, Gemma comparing two short lists of words, took ~25 s, a third of every scan. I first tried asking Gemma only about the staples plain code couldn't place ("is any of these here under another name?"). It was faster but wrong: the 4B model over-claimed, deciding the spinach was the cheddar cheese. So matching is now plain code, with a strict rule: an item counts as a staple only if it ends with it. "Cheddar cheese" is cheese and "free-range eggs" are eggs, but "tomato sauce" is not tomatoes and "milk chocolate" is not milk. Missing a synonym ("OJ") only means buying something you already have; a false match means not buying something you need.

The result, on the same photos and staples:

A scan is now one Gemma call plus two checks that take about a millisecond each:

3. Generation is what takes the time. A recipe request is about 800 tokens; nearly all of its ~30 s is Gemma generating the answer, with model loading and prompt reading taking a few seconds. Shorter answers, not a faster prompt, are what would speed it up next.

4. Checking my own setup. The looping scans' errors never showed up as Sentry issues, even though their traces did. Sentry's usage stats explained why: the error events had been rejected as too large. By default, Sentry's Python SDK attaches each stack frame's local variables to error events, and here those held the prompt, Gemma's answer and the base64 photo, several times over. So errors from real scans were silently dropped, and smaller ones would have sent exactly the data I promised never to send. One setting (include_local_variables=False) fixed both, and I checked it end to end: the same error with a real photo now arrives in Sentry at 11 KB, with no prompt, answer or photo in it.

Privacy is kept, in traces and error reports alike. Only metadata is sent: timings, token counts and how many items each check looked at. The photo, the prompts, Gemma's answers and even the food names never leave the machine, and the tests check exactly that for both.

Gemma 3 4B vs 12B

I also tried the 3× bigger model on the same photos, on the same laptop (before the matching speed-up):

12B reads messy photos a little better, but it's about 3× slower on recipes and the recipes weren't better. I kept 4B. Switching is one environment variable (FRIDGE_MODEL=gemma3:12b) if your machine is faster than mine.

Honest limitations

• It works well on clear, well-lit photos. On messy, dim ones the 4B model still misreads things: it called a pack of beer "juice bottles", and in the demo photo it reads pineapple slices as banana (hence the yogurt & banana bowl).
• It only knows what's in the photo. Something hidden behind the milk shows up as missing.
• Staples are matched by name, so a synonym like "OJ" won't count as orange juice.
• Recipes are suggestions, not nutritional advice.

Why Does Open Innovation Matter?

• A photo of the inside of your fridge is personal. It shows what you eat, drink and can afford. With an open-weight model running locally, it never leaves your laptop. Even the tracing sends only timings and counts.
• It runs on hardware people already own. Gemma 3 4B handles both the vision and the recipes on a 6-year-old M1 laptop. No GPU server, no subscription, no per-request cost, so it can stay free.
• I can see and change everything. I could swap models with one environment variable, constrain the output with a JSON schema that the runtime enforces token by token, cap the answer length, and design the pipeline around the model's real weaknesses. With a closed API I'd be guessing; here I could measure the problem and fix it.
• It works offline, including in a kitchen with bad Wi-Fi.

Prize Categories

• Best Use of Gemma: Gemma 3 4B does the AI work, reading the photo and writing the healthy recipes, entirely locally through Ollama.
• Best Use of Sentry Agent Tracing: every model call and every check on its output is traced (Agent Activity screenshots above). The traces exposed a real bug and showed where the time went, which led to scans 2.6× faster on clear photos. Following up on errors missing from Sentry's stats also uncovered a privacy problem in my own error reporting, now fixed.
• Best Use of ElevenLabs: the demo's voice-over was generated with ElevenLabs text-to-speech (eleven_v4), one line per moment in the video, stitched so it sounds like one take. The script is scripts/narrate_demo.py.

Example photos are from Wikimedia Commons: USDA (public domain) and others under CC BY-SA. Full credits are in the repo.