Sloth Roll 在 iPhone 本機把截圖整理成日曆事件、地點和支出卡片

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

My camera roll is full of screenshots of things I meant to do later. A flyer for a tournament on Saturday. A restaurant someone recommended. A receipt I should have written down. I take the screenshot, and that's where it ends. Later never comes.

So I built an app for that later. Sloth Roll reads my screenshots on the iPhone with an open-weight vision model and turns each one into a card:

• an event goes to Calendar,
• a place opens in Apple Maps,
• a receipt is saved to an expense list inside the app.

I don't type, copy, paste, or switch apps. I only swipe: right to do it, left to skip it. Tap a card to fix the title, date, place or amount first.

It also does the cleanup I keep putting off. Screenshots that are blurry, near-duplicates, or that the model calls not useful end up in a Clean up tab, and I delete them in one tap. They go to Recently Deleted, through iOS's own confirmation, so nothing disappears by surprise.

Everything runs on the phone. After a one-time model download, it works in airplane mode.

The lazy payoff. In one scan of my real library, the app went through 50 new images, read 43 of them with the model, and found 19 actions, in about 3.7 minutes, while the phone sat on the table. My part was 19 swipes. If adding one event or expense by hand takes about a minute, that's roughly 19 minutes of copying turned into under a minute of swiping.

That's my idea of Slow and Steady: the app is the slow one. It reads one image at a time, remembers what it already read, and waits for me. I swipe whenever I feel like it.

Demo

First launch	Scan one album	Pick what to read	Reading and swiping

In the last screenshot, the app is reading image 5 of 27 while the first card is already waiting: a Facebook post that says "MAÑANA TORNEO LIGA SUPER". The card already says what will happen: "Add event · Sat 24 Oct, 3:30 p.m. · Plaza La Cachanilla". One swipe and it's in my calendar.

Code

Sloth Roll

You took the screenshot. Sloth Roll does the rest.

Overview

An iPhone app that reads your screenshots with an open-weight vision model running on the phone and turns each one into a card: an event for Calendar, a place for Maps, or a receipt for your expense list. You only swipe. Nothing is uploaded, and after a one-time model download it works in airplane mode.

Built for the "Slow and Steady" theme: your camera roll fills up with screenshots of things you meant to do later. Sloth Roll does the slow, boring part for you, one image at a time, on your phone, and waits. You don't type, copy, paste, or switch apps. You only swipe, whenever you feel like it.

Demo video: youtube.com/shorts/mwsLE8IcGfI

…

How I Built It

flowchart LR
    UI["SwiftUI views"] --> VM["ViewModels"]
    VM --> SC["ScanCoordinator"]
    VM --> ACT["Actions: Calendar · Maps · Expenses"]
    SC --> PK["PhotoKit"]
    SC --> VN["Vision: OCR, blur, feature print"]
    SC --> EX["Extractor: prompt + parser"]
    EX --> VLM["MLX · Qwen3-VL-2B 4-bit"]
    SC --> SD[("SwiftData")]
    ACT --> SD

It's a SwiftUI app (Swift 6, iOS 18+). The model is mlx-community/Qwen3-VL-2B-Instruct-4bit, a 4-bit conversion of Qwen3-VL 2B. It runs on the iPhone's GPU with MLX Swift and mlx-swift-lm. The download is about 1.8 GB, and it takes about 2 GB of memory while loaded. I tested on an iPhone 17 Pro.

Triage first, model second. A vision model is slow and heavy, so a scan works in two passes. First, with no model loaded, every new image gets cheap checks. Camera photos are skipped (they have camera make, model and lens info in the file). Apple's Vision text recognition must find at least 20 characters, or the image is skipped. The shortest real action in my test set had 40. The same pass measures blur and takes a Vision feature print for finding duplicates, so each image is opened only once. Then the model loads once, reads only the images that passed, newest first, saves each answer right away, and unloads at the end.

Dates relative to the screenshot, not to today. My first prompt resolved "hoy" ("today") against the day I ran the scan, so a cinema ticket from July came back dated October. Now the prompt includes the day the screenshot was taken, plus the next seven days spelled out with weekday names, in Spanish and English. "Mañana", "este sábado" and "hoy" land on the right day.

The model's answer is untrusted. It answers with one JSON object: type, title, date, time, location, amount, currency, confidence. In practice it wraps the JSON in code fences, adds chatter, leaves trailing commas, and once answered with {"error": "..."}. The parser cuts the answer down to the JSON object, checks every field (a real date, a real 24-hour time, an amount that isn't negative), and turns unknown values into safe defaults. A bad answer gets one stricter retry, then the card says "couldn't read this one". It never crashes on bad output.

One small prompt change mattered a lot. One version of the prompt made the model drop the "type" key on 23 of 24 images, so almost every image needed a retry. Listing "type" as a required first field brought that to zero retries.

Order numbers aren't money. The model kept reading order and booking numbers as amounts. The prompt now says amounts must be money (a price or a total) and calls out folio, order and ticket numbers as not amounts.

Clean up reuses the same model. Blur is a Laplacian variance score, with a contrast check so a plain blank screen doesn't count as blurry. Duplicates are screenshots whose Vision feature prints are very close; the newest one is kept. For everything else, the model that's already loaded gets one question: "Is this screenshot still useful to keep? Answer only yes or no." Only a clear "no" puts it in the pile. When in doubt, it keeps the screenshot.

How well does it work? I built a debug eval screen that runs the whole pipeline on an album and lets me mark each row right or wrong. On 19 hand-checked images, the type (event, place, receipt or none) was right on 18 (95%), and all 6 junk images came back "none". The miss was a cinema purchase history read as an event instead of a receipt. The fields are weaker: date and time were both right on 6 of 10 events. The usual slip is the model taking a chat timestamp or the status-bar clock as the event time. That's why every card can be edited before you swipe, and why cards the model wasn't sure about say "not sure". Speed is about 4 to 8 seconds per image once the model is loaded.

The app has 102 unit tests. Services sit behind protocols, so the tests swap in a fake model and a fake photo library.

Why Does Open Innovation Matter?

My screenshots stay on my phone. Screenshots are some of the most private things on a phone: chats, receipts, tickets, addresses, bank apps. A cloud vision API would see all of them. With an open-weight model running through MLX, no image, text or result leaves the phone. There's no account, no analytics and no server.

It works offline. After the one-time download, scanning, cards, Calendar, Expenses and Clean up all work in airplane mode. (Opening a place hands it to Apple Maps, which uses Apple's place search when there's a connection.)

Reading my whole library costs nothing. A scan reads dozens of images, and a camera roll has hundreds of screenshots. With a paid API, every scan would be a bill, and I would scan less. On the phone's GPU, the marginal cost is battery.

I can swap the model. The model ID lives in one file. When a better small open vision model comes out on mlx-community, I change one line and run the eval screen again.

The honest trade-offs:

• The first launch needs Wi-Fi for a 1.8 GB download.
• It needs a recent iPhone with enough memory; I've only tested on an iPhone 17 Pro.
• The prompt and my test set cover Spanish and English only.
• A 2B model gets the type right much more often than the exact time. Swiping is quick, but checking the card still matters.

My Agent Session

I used Claude Code as a coding partner for the Swift code, the tests and the docs. The model in the app does all the reading of screenshots. In the last sessions it prepared the repo for submission, checked that a fresh clone builds, and helped draft this post from the repo's dev log.

Credits and licenses

Model: Qwen3-VL-2B-Instruct by the Qwen team, 4-bit MLX conversion by mlx-community (Apache-2.0). MLX Swift and mlx-swift-lm (MIT). swift-transformers and swift-huggingface (Apache-2.0). App code under the MIT License. Full table in the repo README.