Cal Hacks 13 團隊打造 Statheros，讓用戶以頭部、手掌或語音操控 Krita

What I Built

Statheros is a macOS drawing companion for Krita that lets you draw with your head, your palm, or your voice instead of a mouse or stylus. It’s built for artists who find fine motor control hard: people with tremors, limited hand mobility, or fatigue that makes holding a pen painful.

It runs as a small floating panel next to Krita and does four things:

Camera tracking. MediaPipe face or hand landmarks run locally and map onto a cursor across your display. A five-point calibration (neutral, left, right, up, down) fits the range to how you move, not how a “typical” user moves. It even notices when your camera’s axis is flipped and offers a one-click fix.
A safe drawing area. Statheros reads Krita’s window through macOS Accessibility (no screen recording, no Krita plugin) to find the canvas, and it only presses the pen inside that rectangle. If it can’t identify the canvas, you drag one out with the mouse. The panel and Camera Preview are always excluded, so you can’t scribble over your own controls.
Stroke assistance. You can pick a stroke mode: Natural, Precision, Sketch, or Tremor Filter, which smooths small unwanted oscillations while keeping the stroke you meant. There’s also a “hold still to lift” gesture: once you’ve moved more than 15 points, holding within 7.5 points for 1.2 seconds lifts the pen.
Voice commands. Say “pen down”, “eraser”, “undo three times”, “swap colors”, or just “I want to rub that out”, and it happens in Krita.
What inspired it: drawing tools assume steady hands. Krita’s stabilizer helps once the pen is on the canvas, but it doesn’t help you get the pen there, switch tools, or undo a mistake without reaching for a keyboard. We wanted to remove every step of “pick up the stylus” from the workflow.

What I learned:

Safety beats features. The app starts paused, with camera and mic off and fake input on. Real input is a separate opt-in, and nothing resumes on its own after tracking loss. When software moves someone’s cursor for them, the default has to be “do nothing”.
macOS permissions are a minefield. Ad-hoc signed builds change identity on every rebuild, so macOS silently drops Accessibility, Camera, and Microphone grants even though System Settings still shows them enabled. We fixed it with a stable local signing certificate, after a lot of confusion.
Software you don’t control will lie by omission. Krita doesn’t report whether erase mode is on, so Statheros tracks what it last sent, and “toggle erase” is there to resync.
Latency is a feature. That’s why the voice pipeline is split between local and cloud (more on that below).
Demo

https://www.youtube.com/watch?v=aDrJBGP-vro

Here’s the full loop:

Open Krita and click Set canvas. Statheros finds the drawing viewport and outlines it.
Start the camera and calibrate. Five two-second captures map your range of motion.
Turn on Voice Commands and say “pen down”. The pen arms, then presses as soon as the cursor enters the canvas.
Draw with your head or palm, with Tremor Filter smoothing the line.
“Undo twice, then switch to the brush.” Both steps happen, in order.
Source: https://github.com/ginalzhang/statheros

Partner Technologies

Cartesia: real-time speech-to-text
Voice was the hardest part to get right. Someone who can’t easily reach a keyboard needs commands to feel instant, but they also need to talk naturally, not memorize a vocabulary.

We ended up with a two-tier pipeline:

On-device command words. sherpa-onnx keyword spotting plus Silero VAD runs on a worker thread and listens for Statheros’s fixed command words (“pen down”, “eraser”, “undo”, “bigger”, “tremor filter”, “pause”, and so on). When one is spotted, it runs immediately, even mid-sentence (“switch to the brush”), and that audio never leaves the Mac.
Cartesia Ink for everything else. If an utterance ends without a command word, we stream it to Cartesia’s ink-2 model over WebSocket using the Auto endpoint. A red cloud dot shows on the panel whenever audio is being sent, so you always know when you’re being heard by a server. ⌘⇧Space push-to-talk uses Cartesia’s Manual endpoint for one utterance at a time.
Implementation notes:

We stream only from the start of the current utterance, never earlier, so a command word that already ran locally doesn’t get transcribed and run a second time.
Utterances under 250 ms are ignored, and if more than two seconds of audio backs up on a connection, the utterance fails rather than running a stale command late.
Integration tests run the real VAD and keyword pipeline on WAV fixtures against a local fake Cartesia WebSocket server, including a 401-on-connect case, so we could test failure paths without burning API credits.
What it was like: Ink’s streaming latency was low enough that the cloud path feels natural for conversational requests. Our first design used a “hey cartesia” wake phrase. In practice it was clumsy to say over and over, so we replaced it with open mic plus local command words. That turned out to be both faster and more private.

Anthropic Claude: understanding what you meant
A transcript isn’t an action. “I want to rub that out” should turn on the eraser. “Don’t undo” should not undo. And “undo a few times” needs a number.

Every transcript becomes an ordered plan of actions with repeat counts. Plain phrasings (“undo 3 times”, “undo twice then switch to the brush”) are parsed locally and run immediately. Anything else goes to Claude Haiku, which has to answer using only Statheros’s own action set. It can’t invent commands. That gave us:

Natural-language mapping: “get rid of that last line” → undo.
Negation safety: if the local parser hears a negation, it won’t guess. It defers to Claude, or does nothing if no Anthropic key is configured.
Ignoring non-requests: with an open mic, you’ll talk to people in the room. Claude classifies those as “not a request”, and Statheros ignores them silently, with no error toast.
Bounded repeats: only undo, redo, bigger, and smaller can repeat, at most 20 times, about 60 ms apart.
Only the transcript is sent to Claude, never audio. Command words spotted on-device never call it at all. The API key lives in the macOS Keychain.

Also used
MediaPipe Tasks Vision for face and hand landmarks, running locally in a worker
Electron + React for the panel, and a Swift native helper for Accessibility canvas detection and synthetic pen input
Hackathon Experience

Cal Hacks 13.0, October at the Palace of Fine Arts
FREE YUMMY MATCHA EVERYDAY PROVIDED BY GREPTILE!!!
I built Statheros with Issac and Jedric. I met Issac from a previous hackathon at Dddoski and I met Jedric from my partner's mutual friend. Issac brought in the original “Steady” tracking app, and I built the Cartesia voice input, the Figma design integration, and the Claude intent layer. Jedric had worked on the UI UX design on Figma!

The moment “I want to rub that out” first turned on the eraser by itself, or the first time someone drew a line without touching anything was what I remembered the most.

What’s next: I want to start testing with artists who have tremors, incorporate more Krita commands, and work with other drawing apps!