Aice v0.4.0 如何把本地語音助手部署到酒店、護理院及病房

A voice assistant that runs on your own machine is a nice weekend story. You talk, Whisper transcribes, a local LLM works out what you meant, a skill answers, Piper speaks. No cloud, no account, nothing leaves the room.

Then you try to put it in a hotel room and every assumption breaks.

On your desk there's one user, one microphone, one trusted network, and one person who knows what the assistant can and can't do. A hotel has a hundred rooms, a guest who checks out on Tuesday and a stranger who checks in on Wednesday, a Wi-Fi network shared with every phone in the building, staff who need to see requests instead of hearing them, and nobody who will tell the assistant to "try again" when it goes quiet.

Our last release, v0.4.0, is 29 commits and a few thousand lines of Rust and C++ that deal with that gap. This post walks through what changed and why, including the bugs we only found once real hardware sat on a real table.

The shape of a property deployment

Before v0.4.0 the system was one backend and a desktop frontend. Now a property runs three services plus one small device per room:

 room pod (ESP32-S3)            pod-gateway              aice-backend             facilitator (aice-hotels / -care / -ward)
 mic + speaker + LEDs  ──WSS──▶ speech detection  ──WSS──▶ Whisper STT     ──HTTPS──▶ staff desk, tickets, stays,
 3 buttons                      echo cancellation         LLM classifier             device registry, alerts,
                                Piper TTS, pacing         skills, memory             property MCP tools

• The pod is a microphone, a speaker, a 7-LED ring and three buttons. It does no speech work.
• pod-gateway (the room bridge) turns a stream of audio into turns: it finds where speech starts and ends, cancels the pod's own echo, sends each utterance to the backend, synthesises the answer and streams it back.
• aice-backend is the existing voice core: speech to text, LLM intent classification, skills, memory.
• The facilitator is the property's side: a staff desk, tickets, stays, the device registry, and alerting. It ships as three "packs" with different defaults: aice-hotels, aice-care and aice-ward.

All of it runs on hardware in the building. The voice never leaves the property.

1. Lock the doors before you invite anyone in

The honest first entry in the changelog's Security section: before this release, the desk, the ticket API and the MCP endpoint were reachable without credentials, and a room's turn stream accepted whatever room the client said it was in. Fine on a laptop. Unacceptable on a hotel LAN.

So the first commits are all about trust:

• Staff accounts. The desk now has local accounts with staff and supervisor roles, argon2-hashed passwords, HttpOnly SameSite=Strict session cookies, CSRF tokens on every form and API call, and a five-minute lockout after five bad passwords. Every ticket change goes to an append-only audit log that supervisors can read, and only a supervisor can reopen a closed ticket.
• A service token between backend and facilitator. The pack writes service.token on first run, and the backend presents it as a bearer token.
• TLS from a property-local CA. A new core-tls crate creates a certificate authority for the property, issues server certificates from it and builds rustls configs. On any non-loopback bind the facilitator defaults to TLS, serves its CA certificate for pods to pin, and refuses plain HTTP. The backend and bridge also refuse plaintext on a network bind unless you set allow_plaintext_lan explicitly.
• The room comes from the token, not the client. This is the important one. A pod enrols with its id and a secret nonce. It shows up as pending on the supervisor's desk. When a supervisor assigns it to a room, the pod receives a device token, exactly once. From then on the backend verifies that token on every connection and every turn, and takes the device id and room from the verified token. A client can no longer say "I'm room 214."

Verification is cached for 30 seconds. If the facilitator is down, a recent answer is kept for ten minutes so a desk restart doesn't take every room offline.

2. Memory belongs to a stay, not to a room

Aice has long-term memory (Memory Palace, our palace-rs crate): it can recall what you told it last week. In a home that's useful. In a hotel it's a privacy incident waiting to happen. The guest in room 214 tonight must not hear about the previous guest's allergies.

The answer is the stay. Staff check guests and residents in and out on the desk (or with stay open|close|list). Each stay gets its own memory wing, and a turn from that room reads and writes only that wing: wake-up context, recall, ingestion, and knowledge-graph facts, which are namespaced per wing.

A few details that took more thought than the headline:

• The backend checks the stay on every turn, not on connect. A pod that stays connected through a checkout moves to the next guest's stay instead of keeping the last one. If the facilitator can't be reached, the turn runs with memory off.
• No stay, no memory. A room without an open stay, or any unauthenticated turn in a property deployment, remembers nothing. The shared journal skill is hidden from rooms entirely.
• Consent is recorded per stay. Without consent the pod gets no memory wing at all. Hotels default to consent from booking terms; care homes and wards default to not agreed. Staff tick it at check-in, can change it during the stay, and every change is audited.
• Retention is per property. memory_retention is keep (a returning guest can continue an old stay's memory), archive after N days, or wipe_on_close (the default for wards). A backend job deletes due wings, including their facts, and confirms each purge back to the facilitator.

One thing we learned the hard way while upgrading palace-rs: newer versions record wing names in several new tables (a wing registry, usage events, mined-file records, feedback). A purge that only deleted the memories would have left the names of past stays behind. Purges now clear all of them, so a wiped stay leaves nothing.

3. The pod: from test bed to real hardware

We started with an M5Stack ATOM Echo because it's tiny and cheap. It taught us something important: its microphone and speaker share one clock pin, so it cannot listen while it speaks. That makes it half-duplex. A guest can't interrupt it by voice, which for a voice assistant in a room is a real limitation.

So we wrote down what a room pod actually has to do (full-duplex audio, an echo reference, buttons, LEDs) and moved to the Waveshare ESP32-S3-AUDIO-Board. It has an ES8311 codec driving the speaker and an ES7210 ADC for the mics, both on one full-duplex I2S port. The best part: one ES7210 channel records what the speaker is playing. That loopback is the echo reference that makes everything in the next section possible. The codec register drivers are vendored from Espressif's Apache-2.0 esp_codec_dev.

Two bugs only appeared on the board:

• TLS rejected the property certificate as "not yet valid". The pod had never set its clock, so as far as it knew it was 1970. It now syncs over NTP before opening any TLS connection.
• mbedTLS checks names only, and the property hosts are .local names. The pod now resolves .local over mDNS.

Provisioning without a laptop

A fresh pod derives its id from its Wi-Fi MAC, generates a random nonce and keeps it in flash, and enrols with the facilitator over HTTPS pinned to the property CA. The LED ring pulses slow amber until a supervisor assigns a room on the desk. Then it connects to the room bridge over WSS with its new device token.

If the server replies "unauthorized", the pod clears its token and enrols again. If it lost its token, it can get a new one by proving its nonce, and the old token stops working. Holding the button at boot forgets the identity completely.

Signed updates

firmware keygen|publish|rollout signs images with a property ECDSA P-256 key and offers them to a stable percentage of the fleet. Pods download with their device token and install only an image whose signature verifies. In fairness: that flow is covered by tests, but at the time of the hardware bring-up we hadn't yet run a signed update end to end on the board.

Buttons that work when the AI doesn't

• Long press = help. It raises an escalated help_button ticket for the room, and the bridge speaks a confirmation. No speech recognition or LLM is involved, so it still works when those are down.
• Double tap = privacy mute. The pod stops sending microphone audio and the ring turns magenta. The help button still works while muted.

4. Letting guests talk over the answer

This was the hardest part of the release.

If the assistant starts reading a long answer and the guest says "no, stop, I meant breakfast," it should stop. To do that, the bridge has to hear the guest while the pod's own speaker is playing a few centimetres from the microphone. Without echo cancellation, the loudest thing the mic hears is the assistant itself, and it interrupts itself constantly.

A full-duplex pod now streams two signals during an answer: its microphone and its playback loopback. The bridge's echo canceller does two things.

First, it finds the lag. The echo arrives later than the reference. On our pod the lag measured one 40 ms frame plus 0.8 ms, but whether that extra frame was there changed from session to session. So the canceller runs a correlation search over up to 128 ms every quarter second and keeps the lag only when the correlation is confident.

Second, it learns the echo path with a 256-tap normalised LMS filter (16 ms at 16 kHz) and subtracts the predicted echo from the mic. The core loop is short:

let predicted: f32 = self.weights.iter().zip(window).map(|(w, x)| w * x).sum();
let error = heard - predicted;

let energy: f32 = window.iter().map(|x| x * x).sum();
if self.adapt && energy > ACTIVE_REFERENCE_ENERGY {
    let step = MU * error.clamp(-clip, clip) / (energy + EPSILON);
    for (w, x) in self.weights.iter_mut().zip(window) {
        *w += step * x;
    }
}

The interesting parts are around that loop:

• The double-talk guard. If the filter keeps adapting while the guest talks, it learns their voice as "echo" and starts cancelling them. So every 10 ms block compares mic energy against predicted-echo energy. Echo alone sits near a ratio of 1 once the filter has converged, and a voice on top pushes it up. At a ratio of 2 adaptation stops, and it's judged again every block, so one loud syllable in the answer doesn't freeze the filter for good.
• Robust updates. Even before the guard notices, the first milliseconds of a guest's voice are far larger than the usual residual. Updates use the error clipped to twice its running typical size, so those milliseconds barely move the filter.
• Barge-in needs to beat the echo, not just be loud. The bridge stops the answer only on speech that is loud and clearly above the predicted echo for several frames. The new turn keeps the guest's first words.

Gain staging mattered as much as the maths. The reference input runs at 18 dB so it never clips, the mic has no digital gain, and the speaker is at -20 dB.

Measured on the Waveshare pod: 11–14 dB of echo removed, no self-interruption across 27 seconds of answers, and about 60% of normal-volume speech caught within a second. That isn't perfect, but it's the difference between "you can interrupt it" and "you can't."

One last detail: a full-duplex pod still has a little audio buffered when the answer ends. That tail was being heard as a new request. The bridge now waits until the pod's speaker is actually quiet.

5. Conversations, not commands

Several fixes in this release share one theme: people don't talk in single, clean commands.

• The wake word used to be required on every turn, so follow-ups ("and tomorrow?"), corrections and barge-ins were dropped. Each connection now tracks a conversation: a wake phrase wakes it, and it stays awake while a turn is open, while an answer is playing, and for wake_word.cooldown_secs after playback ends. The bridge reports playback_started / playback_finished, so the window runs from when the guest stopped hearing the answer, not from when the text was generated. We first shipped 2 seconds, which was far too short for a natural follow-up. The default is now 8.
• A pause split one request into fragments. A guest who paused after "Computer, ..." got their request cut into separate turns; one of them arrived as just "of tahini." Now, if speech continues before any answer has come back, it's sent again together with the first part as one request. The bridge also ignores answers for turns it has replaced, so a cancelled turn's empty answer can't swallow the real one.
• Short-term context. The backend keeps the last few exchanges per session for chat replies, and drops them on a new stay or after 10 idle minutes. When a guest interrupts, a playback_interrupted event with the spoken ratio cuts the remembered answer down to what they actually heard.
• Chat was making promises it couldn't keep. Asked for a pillow, the general chat path would happily say "I'll get you an extra pillow right away", and nothing would happen. Chat is now told explicitly that it cannot act, and must say a request couldn't be sent rather than pretend.
• Silence is a bug. When a skill failed (a provider timed out, a place wasn't found), the pod said nothing. Most skill branches were returning their error past the apology code. Every skill now runs inside one block, so every failure is spoken.

Two classifier bugs were especially sneaky:

• Rooms could never make hotel requests. The classifier only offered the hotel skill to clients that declared it, and pods declare nothing. A backend with a facilitator now always offers it.
• The classifier's output was capped at 24 tokens. A hotel request with slots ("two towels to room 214 after 6pm") produced JSON that got cut off, failed to parse, and silently fell back to chat. The cap is now 64, and decoding still stops as soon as the JSON is complete.

6. Making it fast enough to feel like talking

In a voice interface, latency is the UX. A few changes made a large difference:

• Whisper was encoding a full 30-second window for every clip, even a two-second "what time is breakfast." It now encodes only the audio plus a margin: from 4.8 s to 1.0 s per pass on CPU.

• Ollama was reloading the model on every call, about 3 seconds each time. The classifier and memory calls asked for a different context size than the warm-up and the answers, and a different context size means a reload. Every call to a shared host now uses one size, so the model loaded at startup stays loaded.
• STT ran every room through one Whisper context behind a mutex. It now uses a pool of stt.workers contexts.
• Admission control. Turns and STT jobs take permits (service.max_concurrent_turns, stt.workers) and queue beyond them. A turn that waits longer than service.turn_queue_max_wait_ms gets a spoken "busy" reply instead of silence.
• LLM failover. ollama_urls adds hosts next to the main one. Calls rotate across them, and a failing host is skipped for 30 seconds.
• Pacing playback. The pod only buffers about 0.4 s of audio, so the bridge sends answers at playback speed with three chunks of lead rather than dumping them all at once.

A capacity test that can fail the build

room-loadtest enrols simulated pods on a facilitator, then drives them at real-time pace through the real bridge, backend and facilitator, and reports p50, p95 and max from end of speech to first answer audio. In CI it runs eight rooms against a backend with only two turn permits, and requires that every turn is answered, admission holds, and p95 stays under five seconds. If a change makes the system slower under load, CI goes red.

CI also now compiles the pod firmware with PlatformIO, and releases ship Linux x86_64 and aarch64 archives for appliance installs next to the macOS build.

7. When nobody picks up

A request that becomes a ticket nobody reads is worse than no assistant at all. Each pack now has alert tiers and per-tool acknowledgement windows:

• A new ticket pages the first tier.
• If it's still open when its window ends, it's marked escalated (by actor sla, audited) and paged to the next tier, and this repeats on the last tier until someone acknowledges it.
• Channels are webhooks or a tool on the property's own MCP server, which is how you reach nurse call, DECT handsets or PMS paging. A dead channel doesn't block the others, and a first alert that reached nobody is retried every five seconds.
• The schedule lives on the ticket rows, so a restart doesn't reset the timers.

Defaults differ by pack: care homes escalate falls and distress after 60 seconds, wards escalate "get a nurse" after 120, hotels use 15 minutes for everything (and the runbook tells you to tighten that for maintenance and complaints).

Pods are watched too. The backend reports which pods have an open stream every 30 seconds. An active pod that goes silent past device_offline_after_secs raises one escalated device_offline ticket and shows OFFLINE on the desk, because a room that can't call for help is itself something staff need to know about.

8. Finding where a slow turn went

Once real people are talking to it, "it felt slow" is the most common bug report and the least useful one. So the last big change is end-to-end timing of the guest journey.

First, an embarrassing one: every duration metric was exported as a Prometheus summary, so every dashboard panel built on histogram_quantile over _bucket series had been quietly empty. Every *_seconds metric is now a histogram with fixed buckets from 5 ms to 4 hours.

Then the tracing. The bridge times backend wait, speech synthesis, and end-of-speech to first answer audio; the desk times ticket resolution; stage, dependency and LLM durations attach to the turn's trace without touching call sites. Any turn whose wait after speech passes service.slow_turn_threshold_ms (3 seconds by default) is logged once as slow_turn, with:

• the full stage timeline,
• the slowest stage,
• and the time no stage covers.

That last number turned out to be the most useful. When the slowest stage only explains half the wait, the problem is in the gaps: a queue, a lock, a reconnect. It's counted in voice_turn_unaccounted_duration_seconds, and a new "Guest Journey" Grafana dashboard puts it all on one screen.

What we'd tell ourselves at the start

• Decide where identity comes from before anything else. Taking the room from a verified token rather than from the client made memory scoping, alerts and offline detection simple. Everything downstream trusts one thing.
• Privacy is a data model, not a setting. Stays, consent and retention had to be first-class rows, including making sure purges reach every table a dependency adds later.
• Test on the real board early. The clock, mDNS, the shared clock pin and the buffered audio tail were all invisible in tests.
• Silence is always a bug. Busy, failed, muted, can't act: each one now has something spoken.
• Measure the gaps, not just the stages. "Unaccounted time" found more than any single stage timer.
• Write the capacity requirement as a test. "p95 under five seconds with eight rooms" is a sentence until CI enforces it.

What's next

The classifier is now the part that matters most: every request goes through the LLM, and a misclassified request is a missed towel or a missed nurse call. Next we're working on scoring candidate classifier models against a labelled dataset, and running a second model in shadow next to the live one in production to record where they disagree before we switch anything.

About Aice

Aice is an open-source (Apache-2.0), local-first voice assistant written in Rust. Speech recognition, the LLM and memory all run on hardware you own. With v0.4.0 it runs in hotels, care homes and wards: guests ask out loud in their room, and staff and property systems get the request, with no cloud for the voice.

Source: github.com/AncientiCe/aice

v0.4.0 release (macOS arm64, Linux x86_64 and aarch64): github.com/AncientiCe/aice/releases/tag/v0.4.0

If you're running a property and want to pilot it, or you've solved echo cancellation on a microcontroller better than we have, I'd love to hear from you in the comments.