前台接待不必依賴雲端：作者以 942 MB RAM Raspberry Pi 2 打造本地大語言模型接待裝置

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

A front-desk attendant that runs on a 2015-vintage Raspberry Pi 2 — 32-bit ARM, 942 MB of RAM, no internet connection, no subscription.

It does not chat. It sits inside a small menu on a desk display and, at fixed moments, says one short line in Japanese: a greeting when someone wakes it, an acknowledgement when they pick something, a "the workshop is closed now" at the end of the day. The line is generated by a 150M-parameter open-weight Japanese model running locally through llama.cpp.

The people I think would pay for this are not looking for a chatbot. They run a small workshop, a studio, a reception desk, a maker space. They want a screen near the door that feels attended rather than blank. They have one unreliable Wi-Fi router. And they are uncomfortable with the idea that the sounds and words around their front desk end up on a server owned by someone else.

That is the honest product argument: the constraint is the feature. A cloud assistant at the front desk means a monthly bill, a network dependency, and someone else's data retention policy. This thing means a one-time hardware cost, a box that keeps working when the router dies, and a text log that never leaves the building.

It also means I had to make an LLM work on hardware that was obsolete before most people had heard of an LLM. That turned out to be the interesting part — and the reason this post has numbers in it instead of adjectives.

Demo

Demo video: https://youtube.com/shorts/5tsycraxtRs (unlisted)

The footage is a screen recording of the deployment this repository was extracted from: the same board, the same menu, the same pattern — one short line, generated locally. It runs 64 seconds.

1. It opens with the wait: "Waiting for response…" while the local model comes up. Nothing is fetched over the network; the wait is the model being loaded into 942 MB of RAM.
2. The menu then appears with a greeting line already written by the model on the device.
3. Selecting an item asks the local model for a new line, and the terminal shows the result.
4. The interface is Japanese, and there is no narration. This is the device as its operator actually sees it, not a polished screencast.

I left the wait in, and I did not cut a "clean" version. A demo that only shows the happy path is not a demo of something that has to survive a power cut in a workshop.

One honest note on scope: the device in the video runs my existing terminal menu (v2.0.1), not this repository. This repository is the standalone, documented extraction of the pattern — written inside the challenge window — and it is the code I would hand to a customer. The measurements in docs/MEASUREMENTS.md come from that same device.

Code

Repository: https://github.com/sanpeita/offline-attendant

The repo is deliberately small. It is a reference implementation of one idea, not a framework:

• attendant.py — starts the local inference engine, checks it is healthy, and asks it for exactly one line using a fixed template. No user input is ever passed into the prompt.
• config.example.toml — where it lives, what it says, where the model file is, and a loopback-only port.
• systemd/ — a unit example that documents, in comments, why it is configured not to restart automatically.
• docs/BUILD_PI2.md — the build recipe for 32-bit ARM, including the trap that cost me the most time (-j1, -latomic in the right place, --jinja, -c 512).
• docs/MEASUREMENTS.md — every number in this post, with the conditions it was measured under.
• tests/ — 19 tests that need no engine. A fake resident server stands in for llama-server, so the fail-closed paths are exercised for real: unavailable engine, empty output, a slow answer, and a config that tries to bind somewhere public.

If you only read one file, read docs/BUILD_PI2.md. The interesting knowledge on this board is all in the negative results.

How I Built It

The three walls of a 942 MB machine

1. You cannot build in parallel. ninja with no explicit job count picks nproc + 2 = 6 on this board. A single -O3 cc1plus process peaks around 300 MB. One translation unit (common/arg.cpp) took about seven minutes on its own and pushed available memory down to 139 MiB. The build must be -j1. A full build of the completion binary is 70–80 minutes across 73–82 steps.

2. 32-bit ARM needs libatomic, and the link order matters. 64-bit atomics are not inline on armv7, so the executable link fails with undefined reference to __atomic_load_8 and friends. Passing -latomic through CMAKE_EXE_LINKER_FLAGS puts it before the objects, where Debian's default --as-needed discards it. Through CMAKE_CXX_STANDARD_LIBRARIES it lands at the end of the link line, where it actually resolves. Shared libraries link fine either way, which is why the failure only shows up at the very last step and looks like a mystery.

3. Measuring memory can stop the build entirely. My first attempt gated the build on per-process peak RSS from /proc/<pid>/status. On this kernel, VmRSS and VmHWM are simply absent for tasks that have no memory map. A fail-closed gate then cannot distinguish "cannot measure" from "over budget", so it refuses to proceed — and the build ran zero steps. The fix was not to relax the threshold but to move the safety valve to the cgroup v2 memory controller (MemoryHigh, a soft limit; MemoryMax invites the OOM killer) and let the build run detached in a transient unit:

systemd-run --unit=llama-build --collect --no-block \
  --property=MemoryAccounting=yes --property=MemoryHigh=650M \
  ninja -j1 bin/llama-completion

Peak for the whole build: 267 MiB, memory.swap.peak = 0, oom_kill = 0. Measuring the thing was harder than building the thing.

Getting one line out of it

Two obvious options, and the measurements decided between them.

One process per request. Load the model, generate, exit. Clean and simple:

Look at that gap. Generation itself is 1.3–2.0 seconds. The other eight seconds are loading and tearing down the model, every single time. On a desk, ten seconds is not a pause, it is a bug.

Free memory barely moves (~46 MB) even though RSS says 209 MB, because the weights are mmaped and most of that is reclaimable page cache. Swap never moved at all — no thrashing, the board is fine.

One resident process. Start the engine once, talk to it over loopback:

Same hardware, same model, 3.3× faster, because the only thing that changed is that we stopped re-reading the file. That is not an optimization, it is the difference between a product and a prototype.

It also set the design constraint: at ~9.5 tokens/s, a 5-second line budget allows about 40 tokens. So the greetings are 40 tokens and the acknowledgements are 24. I capped the length, not the timeout — a timeout would just turn a slow answer into a missing one.

The real bug was not in the model

Generation was still failing, and the cause was not inference at all. It was an admission gate: "do not use the model unless at least 512 MiB is available."

• Free memory with the desktop running: 473–487 MB
• Requirement: 512 MiB

The gate could never pass. Not sometimes — structurally, every single time, by 40–50 MB. The model had been "broken" for days because a number was wrong.

The tempting fix is to lower the threshold until it passes. That is not a fix, it is a coin flip. The measurements said something better:

• Real drop in free memory: ~46 MB
• Swap: flat

So "512 MiB" had never been measured on this board. It was inherited from a different machine, at a different time, with a different model. I replaced it with a floor derived from what I actually observe — peak RSS plus headroom, 256 MiB — and kept the fail-closed behaviour: if the floor is not met, the line is not generated.

Lowering a number by measuring is engineering. Lowering a number by hoping is how you get a device that works on your desk and not in a shop.

What the real device taught me

All of this was verified locally, and then the first three read-only preflight runs on the actual hardware failed in a row — all three with the same class of defect: a rename that was applied on one side only.

1. A payload key was renamed on the writer, and the reader still asked for the old key.
2. A release-name contract still required the previous version string, so every run on the device stopped on that line.
3. An identifier's group structure changed from three parts to two, and the validator kept requiring the field that no longer existed.

Why did local tests miss all three? Because they checked that a value was present, not that a real value was accepted and a wrong one rejected. Once I fixed the tests to pin one real fixture and assert both directions, the whole class became impossible.

The second lesson was about diagnostics. The device-side bootstrap catches exceptions and returns only the type — a reasonable safety decision that also deletes the message. So the exception body was gone. I solved it by writing a read-only diagnostic that replays the identical byte sequence locally and prints the traceback. Failing safe and failing legibly are different properties, and the second one does not come for free.

The last one is my favourite, because it is pure environment: the preflight requires that swap does not move for five seconds. It failed once, with 600 MB free. The culprit was leftover processes from the stopped desktop session — X server remnants touching swapped-out pages pulled them back in. Not memory pressure: just history. After confirming stillness, the same preflight passed 87/87.

Honest limits

I would rather write these down than have someone discover them.

• A 150M model is not smart. It writes short lines. It gets facts wrong. In device acceptance it confidently wrote "as of 1:55 PM" when the actual time was 17:31. The design requires a correct clock line to be shown separately whenever the model does not state a time, so the screen still told the truth — but a wrong time inside the generated sentence is a new failure mode, and it is next on the list.
• The first line after boot is never generated. A 12-second cold start cannot fit a 5-second line budget. It falls back to a fixed line, by design, and the second line onward is real inference.
• Resident operation with the desktop running is unverified. All resident numbers above were taken with the desktop stopped.
• Long-run thermals and stability are unmeasured. It has not been left running for a week.
• There is no auto-restart, on purpose. If the resident process dies, the attendant goes quiet rather than silently grabbing resources back. For a device in a shop, quiet-and-honest beats alive-and-lying.

Why Does Open Innovation Matter?

This project could not exist as a closed-API product, for four concrete reasons.

1. The unit economics only work at zero marginal cost. Each generated line is a few seconds of one CPU. If every line cost an API call, a front-desk greeter would be a subscription — and a subscription is exactly what this customer is trying to avoid. Open weights plus local inference turns marginal cost into electricity.

2. The hardware constraint is only survivable if you can change the stack. Supporting a 942 MB board is not a configuration option in a closed API. I needed to control quantization (Q8_0 here, Q4_K_M when I want more headroom), context size (512 — and it must be explicit, or the server fails to start on this board), thread count, and whether the model stays loaded at all. That last one was the entire performance story, and it is an inference-server decision, not an API decision.

3. Privacy is structural, not a policy promise. The board has no route to the internet, and the inference server binds to loopback only. There is no outbound port to accidentally enable, no retention setting to get wrong, no vendor to trust. "Your customers' words never leave the building" is a property of the architecture here, not of a terms-of-service page.

4. Legacy hardware is a market, and open source is the only one serving it. There are millions of perfectly good 32-bit boards sitting in drawers. A closed stack would tell their owners to buy new hardware. llama.cpp builds on armv7 if you know about -j1 and -latomic, and the model is a 157 MiB file. That is a product someone can install in an afternoon and understand completely — including the parts that are slow, because you can read the source of the thing that is slow.

The open part is not a checkbox on this build. It is the reason the numbers in this post exist at all: I could measure the thing down to its page-cache behaviour because I own every layer of it.

Prior Work and Credits

Being explicit, because the rules ask for it and because it is the truth.

The build knowledge in this post — the -j1 requirement, the -latomic link-order trap, --jinja for the Japanese template, the 12-second cold start, the 3.096-second resident line — comes from my own private lab notes from earlier in October, where I first got llama.cpp running on this specific board and later measured resident mode against one-process-per-request.

The repository submitted here was written during the challenge window (October 17–18, 2026) and contains no code copied from those notes; what crossed over is the measurements and the lessons, which are reproduced in docs/ with their conditions attached. The prior work was not public, so nothing here is derived from someone else's project.

Standing on: llama.cpp (MIT) and an open-weight Japanese model from the llm-jp project (150M instruct, Q8_0 GGUF, 157 MiB). Neither was modified.

What I Would Charge For

A one-time install, not a subscription: the board, the display, the flashed image, and a configuration pass where we sit down and write the lines the shop actually wants to say. Support as an annual option, because the failure I would expect in the field is not the model — it is a power supply.

The next three things I want to fix, in order: stop the model from stating a time it cannot know; get the resident path verified with the desktop running so a normal machine works, not just a stripped one; and measure what happens after a week of being left on.

If you have a spare Pi 2 in a drawer and a front desk that feels too quiet, the build notes are the part worth stealing.