Ghost Visitors 以本地 Gemma 4 E2B 模擬訪客並檢查網站問題

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

What I Built

Every website has visitors who leave without saying why.

Marisol tried to order on the bus. The page took over three seconds to load, came out wider than her phone, and the plus button was smaller than her thumbnail. Tomás read the web by ear. His screen reader said "textbox" with no name, and the "Place order" button never got focus, so for him it didn't exist. Abuela Rosa set the ofrenda every November. Even at 200% zoom, the light grey text was too faint to read.

None of them filed a bug report. They just closed the tab.

Ghost Visitors brings them back. Each one is a ghost: a persona driven by Gemma 4 running locally, wearing a real browser set up like their device. A ghost gets a goal ("order some bread for the Day of the Dead"), browses your site the way that person did, and leaves a haunting report in their own voice: a verdict, a tombstone epitaph, and what scared them away.

That's how I read this week's theme. Día de los Muertos isn't about fear. It's about remembering people and setting a place for them. Ghost Visitors sets a place for the visitors who quietly gave up.

One rule keeps it honest: Gemma speaks, code checks. A small model will happily invent complaints. So the browser measures things (load time, page width, tap target sizes, accessible names, font sizes, contrast), and a complaint only counts as a haunting if it cites a measurement of the same kind. Everything else becomes a whisper: still shown, but marked unproven, with the reason it was set aside.

Demo

Here is Marisol's report from the broken demo bakery. Three hauntings, each with the measurement that proves it:

And here is why the guard exists. On the fixed bakery, Gemma still had Marisol call the site "a nightmare" with "tiny buttons". Nothing the browser measured backs that, so both complaints become whispers:

The best moment was Tomás, keyboard only, on the fixed bakery. These are his real steps from the eval:

 6. [/order] I need to select a bread item first.
      → pressed Tab; focus is on [3] combobox "Bread"
 7. [/order] The bread selection is open, I will choose the Pan de muerto.
      → typed 'Pan de muerto ($6)' into [3] 'Bread'
 ...
11. [/order] The screen reader is focused on the Email field; I will input an email and then submit the order.
      → typed 'tomas@example.com' into [5] 'Email'
12. [/order] The email is filled, now I must press the place order button.
      → pressed Enter; now on /thanks

On the broken bakery, the same ghost heard textbox (no name announced) where the email field should be. Then he tabbed past the quantity buttons, and focus fell off the end of the page, because "Place order" there is a clickable div. He circled for 32 steps and wrote his own epitaph: "Here lies Tomás, who ordered bread but never found the 'Place Order' button."

Try it yourself (the model download is about 4 GB):

ollama pull gemma4:e2b
pip install -e . && playwright install chromium
ghost-visitors bakery --mode broken &
ghost-visitors haunt http://127.0.0.1:8001 --open

Or run ghost-visitors web for a small local UI where you pick a URL, the ghosts, and their goal.

Code

Ghost Visitors 👻

Persona ghosts, driven by a local open-weight model (Gemma), haunt your website and report what scared them away.

Every site has visitors who quietly leave and never say why: the commuter whose phone shows a page too wide for the screen, the blind visitor whose screen reader announces "edit text, blank", the grandmother who can't read faint grey text even with her glasses on. Ghost Visitors brings them back as ghosts. Each ghost browses your site the way that person really did, decides where to go with Gemma, and writes a haunting report in their own voice.

The rule that keeps it honest: Gemma speaks, code checks. A complaint only counts as a haunting when it cites something the browser actually measured. Anything else is shown as a whisper: kept, but marked unproven.

On the fixed bakery, Gemma still had Marisol call the site "a nightmare"…

MIT licensed. Python, Playwright, Flask, and any OpenAI-compatible endpoint (Ollama by default). The repo includes the demo bakery, 46 tests that run a real browser without a model, and every raw eval run.

How I Built It

The three ghosts

Each ghost has a body and a lens. The body is a Playwright browser context: viewport, device scale factor, touch, and network throttling through the Chrome DevTools Protocol. The lens is the short list of things the browser measures for that person. Tomás's body is the strictest. Controls that Tab can't reach are left out of what he perceives, because for him they don't exist.

One step of a visit

1. Observe. A script scans the page and returns controls with their accessible names, the focused element, visible text, and measurements. Each new measurement gets an evidence id: E2 horizontal_scroll on /: page is 981px wide on a 375px screen.
2. Decide. Gemma reads that perception as the persona and answers with one JSON action: click, fill, select, scroll, back, or for Tomás tab, shift_tab, type, enter, space. Each answer also has a thought and a "fright" score from 0 to 10.
3. Act. Playwright performs the action and tells the ghost what a person would notice: the form was not sent, the browser points at [5]: 'Please fill out this field.', or for Tomás, focus is on [4] textbox "Your name".

When the visit ends, Gemma writes the report and cites evidence ids. Then guard.py judges each complaint:

cites nothing                      → whisper: "cites no measurement"
cites an id that doesn't exist     → whisper: "cites E9, which was never measured"
complains about something unmeasurable → whisper: "'other' isn't something the browser can measure"
cites evidence of another kind     → whisper: "cites E1 (horizontal_scroll), which doesn't show small_tap_target"
otherwise                          → haunting, with the measurement attached

The model never decides what counts as proof. It can only point at it.

Making a 2-billion-parameter model behave

Everything ran on Gemma 4 E2B through Ollama on my laptop, which has no GPU. Each call takes about 13 seconds, so a ghost's visit takes three to twelve minutes. Small models don't follow formats strictly, so most of my work went into meeting Gemma halfway:

• Gemma has no system role, so the instructions are folded into the user turn.
• It sometimes nested the action ({"action": {"action": "click", "target": 3}}), used the action as a key ({"fill": "4"}), or wrote "click [7]". One normalize() function accepts all three.
• When I reused the browsing prompt for the final report, it answered with another browsing step. The report now gets its own persona-only prompt.
• It quoted whole evidence lines ("E1 goal_not_completed on /order: ...") instead of "E1". The guard now pulls the ids out of whatever it quotes.

The eval, and what it caught

The repo ships Marigold Bakery, a tiny Día de los Muertos shop with two modes. broken has seven everyday problems: a render-blocking slow script, a fixed 980px layout, an email field with only placeholder text, 16px quantity buttons, a "Place order" that is really a clickable div, 11px body text, and light grey text on white. fixed has none of them.

eval/run_eval.py sends the ghosts to both sites and commits every raw run.json. eval/score.py re-scores those files offline, re-running the guard on Gemma's raw complaints, so anyone can check the numbers without a model. "Gemma alone" counts everything Gemma complained about. "Gemma + guard" counts only proven hauntings.

Every planted problem was found in every run, and across all runs the guard let through exactly one complaint about a problem I hadn't planted. That complaint was right.

Marisol found a bug I didn't plant. In run 1, on the "fixed" site, she complained that "Back to the breads" was too small for her thumb, and cited a measurement: 130×19px on the thank-you page. My own tests never visited that page. I added it to the tests, watched them fail with her exact finding, and fixed the link.

The guard had a bug too. Tomás honestly reported that he never finished, but quoted the whole evidence line, and the guard rejected it as unmeasured. Because score.py re-runs the guard on saved complaints, I could fix it and re-score run 1 without calling the model again.

Tomás couldn't hear. He ran out of steps on both sites in runs 1 to 3, even the fixed one, so I read his steps. After a page load nothing had focus, so his first type went nowhere. Pressing Enter on a link to the current page silently reloaded it. Pressing Space on a dropdown opened a headless popup that swallowed every later key. A real screen reader announces all of that, and my ghost heard none of it. Once he could hear focus changes, reloads and dropdowns, and got twice the steps (every Tab is a step), he ordered on the fixed site in 13 key presses and failed on the broken one for exactly the planted reason.

No ghost ever claimed success without reaching the confirmation page.

This is one small model on one small site, so read it as a sanity check, not a benchmark. The point it makes is narrow but useful: in each full run Gemma alone raised 3 to 5 false alarms, and the guard kept every real finding while dropping all of those false alarms.

A ghost on every push

The same check runs in GitHub Actions. The workflow runs the tests, installs Ollama on the runner, pulls Gemma 4 E2B, and sends Abuela Rosa to both bakeries. The broken one has to be haunted (--fail-on-haunting exits with code 3) and the fixed one has to rest in peace. Her report goes into the job summary. A run takes 7 to 10 minutes on the runner's CPU, and it needs no API key and no secret, because the model is just a download. Pointing it at your own site is one line:

ghost-visitors haunt http://127.0.0.1:3000 --ghosts rosa --fail-on-haunting --summary "$GITHUB_STEP_SUMMARY"

The session behind it

I built this with an AI coding assistant, and the session is saved below. Here is the part where the eval runs caught Marisol's bug, the guard's bug, and Tomás's deafness:

And the whole build, from "can I enter this challenge?" to the CI ghost (tool outputs left out, local paths scrubbed):

Honest limits

• The checks are inspired by WCAG (24px targets, 4.5:1 contrast), but they aren't a full audit. Rosa's 200% zoom is a smaller viewport with a 2× device scale factor, which is close but not identical.
• E2B is small. A larger Gemma should make Tomás steadier, and switching is one flag: --model gemma4:e4b.
• On a CPU, a full haunting takes minutes, not seconds. Good for a coffee break or a CI job, not a keystroke.

Why Does Open Innovation Matter?

The site you test never leaves your laptop. Ghost Visitors is most useful before launch: on a staging site, an intranet, or a client's half-finished shop. Those are exactly the pages you shouldn't stream to a third-party API. With Gemma on Ollama, screenshots, form contents and page text stay on your machine. Every visit in this post ran with no cloud model involved.

A run costs nothing, so you can run it again. Four eval runs, 20 ghost visits, a few hundred model calls, and I never thought about a bill. That's what made it practical to read a failed run, fix one thing, and run it again. It's how Marisol's bug and Tomás's deafness were found.

CI without secrets. A closed model in CI means storing a paid key in the repo's settings and paying for every pull request, including ones from strangers. An open model is a 4 GB download on the runner. Anyone who forks the repo gets a working haunting check for free.

Open weights made "don't trust the model" practical. The model is a swappable part behind an OpenAI-compatible URL, so I could build the whole system around checking it. The guard works the same with a 2B model on a CPU or a hosted Gemma, and the eval shows exactly what the guard adds for each.

It won't disappear. The model is a file on my disk and the code is MIT. If a service shuts down next year, the ghosts keep haunting.

Prize Categories

Best Use of Gemma. Gemma 4 E2B is the mind of every ghost. It decides each step from the ghost's perception, keeps the persona's voice, and writes the verdict, epitaph and complaints. It runs locally through Ollama and on a GitHub runner's CPU, and the same code works with hosted Gemma through any OpenAI-compatible endpoint. The eval compares Gemma's raw output with the guarded output, so you can see where a small open model is reliable and where code has to check it.

Best Use of GitHub Copilot (GitHub Actions). Actions turns a haunting into a check that can fail. haunt.yml runs the 46 tests, then a two-job matrix installs Ollama on the runner, pulls Gemma 4 E2B, and sends Abuela Rosa to the broken and the fixed bakery. Each job asserts the exit code it expects: 3 (proven hauntings) on the broken site and 0 on the fixed one, so the workflow checks that the ghost can tell the two sites apart, not just that it runs. Rosa's report goes into the job summary with the measurement behind each haunting, and her screenshots are uploaded as an artifact. The first run passed with no secrets configured: tests in 1m13s, the broken bakery in 9m4s, the fixed one in 6m36s. Runs replaced by a newer push are cancelled, and pushes that only touch docs are skipped, so CPU minutes aren't wasted on a model that runs on every push.

Credits

The idea grew out of Ghost Customers, my GitLab Transcend hackathon project, where scripted personas walk every build in CI. Ghost Visitors is a new project, built this week from an empty repo. Here an open-weight model drives the browser, it can be pointed at any site, and its claims are checked against measurements.

I built Ghost Visitors with an AI coding assistant (Cursor) helping me write and test the code and draft this post. Every run, number and quote above comes from the committed eval files.