Snap Me：結合 Gemma 4 與瀏覽器語音，從身邊物件學英文

In Libreville, private English lessons are out of reach for many families, and word lists need a discipline that few beginners keep. I teach English, and I know one thing works better than lists: a word stuck to an object you actually own.

So I built Snap Me in about a day for the Hacktoberfest Hack Day Libreville 2026 (Microsoft User Group Gabon, theme "AI belongs to everyone"). You photograph an object, and the app gives you its English word, its pronunciation, three example sentences and two fill-in-the-blank exercises. A voice says the word out loud. Every exercise you get right earns XP, which unlocks the past and then the future tense.

Try it: https://snap-me-omega.vercel.app
Code: https://github.com/Nelge-3D/snap-me

Two open models, no backend

The whole AI chain relies on open models:

Gemma 4 looks at the photo and writes the lesson. I call it through the Gemini API, with the key kept on the server.
Kokoro-82M generates the voice. It runs entirely in the browser through Transformers.js, so no audio ever leaves the device, and the app falls back to speechSynthesis if Kokoro is not available.

There is no database and no account. Progress, XP, the daily quota and the notebook all live in localStorage. That was a constraint I chose to stay shippable in a day, and it turned out to fit the use case well.

Making Gemma 4 fast enough

My first version took 9 to 13 seconds to identify an object and 34 to 66 seconds to generate the lesson. Nobody waits that long after taking a photo. Two changes fixed it:

Split the work in two calls. The first call only sees the image and returns a few tokens (word, French translation, pronunciation), so the word appears right away. A second, text-only call then writes the sentences and exercises in the background.
Turn off the model's internal reasoning. The documented thinkingBudget: 0 returns a 400 error on this model. I found thinkingLevel: MINIMAL on a developer forum instead:

const FAST_CONFIG = { thinkingConfig: { thinkingLevel: ThinkingLevel.MINIMAL } };

With it, identification takes about 3 to 4 seconds and the lesson about 5 to 6 seconds, and the JSON stayed valid on every call I made. One caveat: this parameter is not in the official documentation, and I only validated it on a handful of calls, so I treat it as a tuning I would re-check before relying on it.

Designed for slow, expensive data

Photos are resized in the browser before anything is sent: a 512px version goes to the API, and a 768px version is kept for the on-screen result and the notebook. Sending a multi-megabyte photo on every snap would be a product bug where mobile data is costly. For the same reason the voice runs locally instead of calling a server on every listen, and the app is limited to 5 photos a day, which also nudges learners to pick an object that matters to them.

What small models teach you

Two bugs from real use, both fixed with prompt changes:

A photo of a team around their laptops came back as "laptop". My prompt told the model to prefer objects over people. Now, when a photo is mainly several people, it names the group ("team", "family", "friends") and keeps preferring an object for a single person.
One run returned the pronunciation of the French word instead of the English one. The prompt now says the pronunciation is for the English word, with examples, and the app always displays it between slashes.

The lesson: a small model has no guaranteed structured output, so I validate every field it returns and fix the format on my side instead of trusting it.

What's next

Exporting and importing progress to change device, recognizing several objects in one photo, and a spaced-repetition review mode for the notebook.

AI disclosure

I built this project with the help of Claude Code (Anthropic), and I directed the product, reviewed the changes and tested the app on real photos. Gemma 4 and Kokoro-82M run inside the app. The design reference mockups in the repository (docs/references) were generated with AI and only served as design inspiration, they are not used in the app. The logo, icons and share image are drawn in code, and the screenshots are real captures of the app. I drafted this post with AI assistance and edited it myself.