Couch Commander：完全離線、支援 Pidgin 的 YouTube 語音遙控器

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Everyone loves a good alone time. It could be that moment where you're lying down, the show is playing, someone says something important, and you miss it. The laptop is across the room. The remote doesn't exist because it's a laptop.

So I built Couch Commander. I say "Hey Jarvis", a little pixel sloth pops up in the corner of the screen, and I tell it what I want, the way I'd actually say it:

• "abeg go back small" → rewinds 15 seconds
• "I no hear wetin she talk" → rewinds 10 seconds
• "make am louder" → volume up
• "skip this one" → next video

It controls YouTube in Chrome on my Mac. No remote, no keyboard, no getting up. Everything runs on my laptop: no internet, no account, no API key.

Demo

Both runs in the video are real recordings at normal speed, not edited.

Code

Bennykillua / couch-commander

Control YouTube from the couch with your voice. Wake word + Whisper + Gemma 4, running fully offline on a Mac. Understands Pidgin too. 🦥

Couch Commander 🦥

Control YouTube from the couch with your voice. No remote, no getting up, no internet.

Say "Hey Jarvis", wait for the chime, then say what you want, in English or Pidgin:

• "abeg go back small" → rewinds 15s
• "I no hear wetin she talk" → rewinds 10s
• "make am louder" → volume up
• "skip this one" → next video

A little pixel sloth pops up in the corner to show it heard you.

Why it runs fully on my Mac

An always-on mic in your living room only makes sense if nothing leaves the room. Every piece runs locally:

No account, no API key, no cloud.

How well the brain works

I tested…

How I Built It

Five open-source pieces are chained together. Each one does one job:

The brain is the interesting part. Gemma gets my sentence and has to return strict JSON, one command from a fixed list or none:

{"command": "rewind", "value": 15}

I set the temperature to 0, used Ollama's JSON mode, and added Pidgin examples to the prompt. If the answer isn't valid, it falls back to none. That rule matters more than anything else in this project. Pausing your movie for no reason is worse than doing nothing.

The numbers

I tested 30 phrases on two local models. 22 of them were phrases the model had never seen as examples, including Pidgin I made up on the spot ("carry am go front small", "I wan hear am well well").

Two things surprised me:

Thinking mode was killing the speed. My first run of Gemma 4 averaged 1,489 ms per command. Turning off thinking ("think": false) brought it to ~190 ms: 8x faster, with no drop in accuracy. For a remote, nobody needs the model to reason. They need it to answer.

How a model fails matters more than how often. Gemma 4's two misses both returned none, so it just didn't act. Gemma 3's misses were actions: it paused the video when I said "tell me a joke", and when I said "make we go back two minutes" it went forward two minutes. That's the difference between a tool you keep and one you uninstall.

Things that broke

• I started with the Apple TV app. AppleScript keystrokes didn't reach the player, so I switched to YouTube in Chrome, where I can run JavaScript directly on the <video> element. Much more reliable. You need to turn on View → Developer → Allow JavaScript from Apple Events in Chrome.
• Whisper kept transcribing my commands as "" or "if you do". The audio was raw 16-bit numbers, and Whisper wanted floats between -1 and 1. One line fixed it.
• Most of the delay wasn't AI at all. It was a fixed 4-second recording window. Wake word took 0.13s, Whisper 0.3s, Gemma 0.25s, and I was waiting 4 seconds for a timer. Voice activity detection fixed that.

Why Does Open Innovation Matter?

Because this thing has a microphone that's always on, in my living room.

With a closed API, every "Hey Jarvis" would send audio from my house to someone's server. I wouldn't use that, and I wouldn't ask anyone else to. Here, the audio never leaves my Mac. You can unplug the router, and it still works.

Open models also let me measure and swap. I ran the same 30 phrases through two models, compared accuracy and speed, and switched off a feature (thinking) that was slowing me down. That's how I found the 8x. With a closed API, I'd get whatever latency it gives me.

And it costs nothing to run. There's no per-request bill for telling my laptop to rewind.

The Lazy Payoff

I used it on my own couch while building it this week. The honest version: it works best from a few meters away. The laptop mic struggles if I'm far across a noisy room, and Pidgin is harder for the small Whisper model than plain English. My next step is using AirPods as the mic, since they're right by my mouth.

But the core loop works: "Hey Jarvis", sloth pops up, "abeg go back small", it goes back. I didn't move.

Prize Categories

• Best Use of Gemma: Gemma 4 E2B is the brain that turns English and Pidgin into commands, running locally through Ollama. I benchmarked it against Gemma 3 4B, and tuning it (thinking off) made it 8x faster.

Built with openWakeWord, webrtcvad, Whisper, Gemma 4, and Ollama. MIT licensed.

The sloth is original pixel art for this project. 🦥