Halal Food Scanner 手機成分辨識復盤：裁剪故障、模型測試與 AI 不負責判定的原因

I build Halal Food Scanner, an iPhone and Android app (Expo / React Native) that tells you if a food is halal. You scan the barcode, and when the product isn't in Open Food Facts or has no ingredient list there, you take a photo of the ingredients instead. The app answers Halal, Haram or Uncertain, with the reason for each ingredient.

The photo path is where most of the hard work is. This is what we learned building it, including a bug I shipped myself.

The model reads, a table rules

The first decision was the most important one: the AI does not decide if something is halal.

A photo goes to a vision model in one call. The model has one job: find the ingredient names on the pack and map each one to an id in our own table ("Gelatine" -> gelatin, "E471" / "mono- and diglycerides of fatty acids" -> e471, "Lab" -> rennet). The verdict comes from the table, not from the model.

The table is a human-reviewed list of 401 E-numbers (261 halal, 138 uncertain, 2 haram) plus named ingredients like gelatin, rennet, whey and natural flavourings. Each row is sourced to halal bodies and standards (JAKIM MS 1500, IFANCA, SANHA, MUI and others). Where they disagree, the app follows the most cautious reviewed position and shows the others.

Two rules fall out of this:

• If an ingredient can come from a halal or a non-halal source and the pack doesn't say which (gelatin, E471, whey), the answer is Uncertain, not a guess.
• If the model finds a word that isn't in the table, the answer for that word is Not checked. A gap is never shown as a pass.

Why so strict? Because when we benched models on real photos, some of them invented ingredients. On a 30-photo bench of Japanese and French packs, one model added ingredients that weren't on the pack at all. A small, fast model did the same on other labels. In a halal app an invented ingredient is a wrong answer, so the model only gets the part of the job where a mistake is visible and checkable: reading.

We now send photos to Gemini Flash first and only fall back to other models when it fails or times out. For blurry, sideways Arabic labels, an empty read gets one retry with the stronger model.

The crop that cut the ingredients out

At the time, ML Kit ran a quick text check on the phone before we uploaded anything. It told us if the photo had readable text at all, so we could ask for a retake right away instead of after a network round trip.

On October 15 I made what looked like a smart change: ML Kit already knows where the text is, so crop the upload to that text box. Smaller image, bigger letters, better reads. We were sending the photo at 1024 px on the long edge, so the crop should help small print.

It didn't. After the change, the share of ingredient photos that came back with no ingredients went from about 40% to 48%.

The reason only showed up when I looked at the photos. Since mid-October the app keeps failed ingredient photos for 30 days in a private bucket, so we can check failures by hand. Two things were in there:

1. Sideways photos. People hold the pack however it comes off the shelf. On a sideways photo, the box ML Kit drew was often around the brand name or the nutrition table, and the ingredient list was outside the crop. The model got a clean picture of the wrong text.
2. Our own test shots. A good chunk of the "empty" photos were us: a desk, an icon, a hand. Without looking, I would have blamed users for bad photos and moved on.

The fix was to stop cropping. The whole frame now goes up at 1600 px instead of 1024. On a bench of 8 full-resolution Open Food Facts photos, 7 read the same and one small multilingual label went from 4 lumped strings to 18 separate ingredients. The cost is about 0.8 s more per read (median ~3.8 s -> ~4.6 s), which is fine for a photo you take once.

Then we removed the on-device check too

Two weeks later the same kind of look at the data killed the ML Kit check completely. A third of ingredients scans took 2 or 3 shots because ML Kit judged the photo unreadable, and Arabic and Japanese labels always used all three, even though the cloud model reads them fine. Median capture time was 4.6 s with retakes against 1.9 s for one shot. Now the app sends the first photo straight away, and the native module is gone from the app.

What the user sees when it fails

The old app showed an error alert when a photo came back empty. Now it shows a small picture sheet: wrong side of the pack, too blurry, and from the first miss a button to scan the barcode instead. Showing a picture of a good ingredients photo works better than any sentence we wrote.

This matters more than it looks. People whose first day gave them an answer came back on another day 38% of the time. People who got no answer came back 10% of the time.

What I'd tell anyone reading text from photos with an LLM

• Keep the model's job small. Let it read and map to ids you control. Put the decision in data you can review and test.
• Unknown is an answer. "Uncertain" and "Not checked" are better than a confident guess, and users understand them.
• Keep failed inputs and look at them. With care (private storage, short retention), but look. Our metric said users take bad photos. The photos said I shipped a bug.
• Be careful with optimizations that trim the input. A crop, a downscale, a token limit: each one can quietly remove the thing you needed.

If you want to see the rulings the table uses, they're public on the site, for example is E471 halal? and the full E-number list. The app is at halalfoodscanner.app.