8 GB 筆電上的小型本地 LLM 會議評測：任務負責人錯判與程式碼補救

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

I used this week's DEV challenge to make a product I had been thinking of making for some time now: a voice-based meeting assistant. I had made some earlier versions and prototypes, but most of them were wrappers around a bunch of LLM API calls that orchestrated the flow. I was burdened by two dependencies: the model's availability on the free tier, and the cost of using it for actual meetings.

Both had been hurdles. But Hacktoberfest has brought me not just a solution to my problem, but an opportunity to improve myself towards the role I aspire to.

When I read the challenge parameters, even the general ones that require open models excited me to be part of such a community and competition. But when the special theme dropped, I thought it was a sign from heaven. Slow and Steady, wow. It was perfect for me and my laptop. I had to wait 5 seconds for the challenge page to load, and then I saw that they were actually referring to my laptop. It made me super excited and determined to complete my project.

But this time, I did not just wrap something; I made research my primary focus. Research into which open model was right for my slow but steady laptop: which model made the best of the limited resources my laptop provides, to give a steady performance. And I was very glad to find the actual parameters that matter here.

So in this post (and a research paper I am planning), I will explain in detail the meeting assistant I built, which is powered by local models and runs steadily even on an 8 GB RAM laptop. Yes, it is comparatively slow, but it is a steady tool that provides cost independence, data privacy and, most importantly, complete customization. One could even fine-tune their own personal assistant, right down to the binaries (future work).

In short:

Just click your name and speak: the app transcribes each turn and writes the action items (owner, task, deadline) as sticky notes, one color per person. A white card means nobody has taken that task yet, and a red check marks anything the code is unsure about. You decide. Audio and notes never leave the laptop.

Demo

This video shows a short demo of the app and how to use it. We tested it in an actual meeting with my friends, planning a trip, but for a proper demo the video was recreated from a script based on that meeting.

A scripted re-enactment: screen-recording a live meeting wasn't feasible on this laptop. The voices are synthetic (Kokoro); the transcription, notes and timings are real, from one recorded take.

Code

I have shared the eval harness and the app code in the public repo.

Meeting Assistant: local, on an 8 GB laptop

A privacy-first meeting note-taker that runs entirely on a CPU-only, 8 GB laptop. Click your name talk; it transcribes with Moonshine, turns the meeting into owner · task · deadline with a small open model (Ternary Bonsai 4B by default), and then checks the model's work in plain code Audio and notes never leave the machine.

Behind it is an eval of three small open models on this exact laptop, because the first question was not "how do I build it" but "which model is good enough here, and what does each choice cost?"

Demo video: youtu.be/XEpmtArXQPA (70 s, a scripted re-enactment; see below) · Write-up: link coming (DEV)

What it does

Speaker buttons: click your name, speak, click the next person. Exact names, no guessing.
Speech-to-text on CPU with Moonshine, about 10× faster than real time here.
Action items…

Setup is in the README. The eval method is in eval/README.md, and how the video was made is in demo/.

How I Built It

Open-source AI used (all local, CPU only):

Moonshine for speech-to-text.
Ternary-Bonsai-2-4B, run with PrismML's llama.cpp fork (the app's default model).
Granite 4.2 3B and Phi-4-mini, run with Ollama.
FastAPI for the local web app.
Kokoro for the demo voices only.

The setup:

A Ryzen 3 4300U with 8 GB of single-channel RAM and no GPU.
8 test meetings with 53 action items that I wrote by hand before any model ran, plus 2 held-out meetings for the second round.
A harness that logs quality, speed, tokens and RAM for every run.

The limited resources I had forced me to do something crucial that everyone should do: research what a model actually does, what is needed and what is not. Below is what I found.

What I found:

The speed limit is arithmetic. The numbers on paper didn't hold up in testing. The smallest model, Bonsai, should have been about twice as fast as the others; in practice its whole runs took longer than Phi's (36 s vs 27 s). Memory bandwidth (21.3 GB/s) divided by model size gives a ceiling on speed. Granite and Phi reached 72–73% of theirs. The 2-bit Bonsai reached only 45%, probably because it is compute-bound. The impact of compute is something that I was able to infer thanks to this study.

"Thinking" made it worse. I tested every model with thinking mode on and off, changing nothing else. With thinking mode on, Granite's recall fell from .94 to .28 and a run took 454 s instead of 44 s. 7 of 16 thinking runs hit the 3,000-token limit with no answer. On a small machine, "think harder" isn't free. We need special optimizations and fine-tuning at different levels, which is planned for future work.

Finding tasks is easy; owners are hard. All three models found 91–94% of the tasks. But "we need a venue" went to the speaker, Akhash, in 5 out of 5 baseline setups. A better system prompt cut invented deadlines from 6 to 0 (Phi and Bonsai), but it didn't fix owners. What these models miss is the subtle part that needs the whole picture: who actually took the task. To solve this, we use helper programs.
About ten lines of plain code fixed owners. Rule: if someone says "we need…" and never takes the task, the task is unowned. This fixed 11 wrong owners with 0 false alarms. (I wrote the rule after seeing these meetings, so it still needs testing on new ones.) The helper programs act as guardrails that enforce what is hard to get through the prompt alone with small models (SLMs).
Real use found what the eval couldn't. The first demo take lost the first half-second of every turn ("We need a hotel" → "Tell near"). The fix was one microphone stream per meeting, with a test that fails on the old code. Lesson: test the whole pipeline with audio, not just text. At the end of the day, actual implementation in an application is where new variables are introduced and one has to tackle them properly.

So in the end, to reach amaze, amaze, amaze, one has to enhance, enhance and enhance.

Why Does Open Innovation Matter?

Open innovation gives one an opportunity to improve not just oneself but also a community. And I am happy to share that I was able to build something because of such open innovation: a meeting assistant that is powered by local models. I did not just use them out of the box, but made an eval harness that compares the available options. This eval harness is my small contribution back to the open innovation without which I would not have been able to build this.

The use of open models also keeps the data on my laptop, and removes the cost dependency.

My actual laptop is very basic, and that is why I was able to research how I can make the best out of the resources I have. This sent me on a journey of understanding what the exact parameters and aspects are that affect the models, and how I can use them. The impact of RAM size is the most fundamental, but I was able to see that one specific model (Ternary-Bonsai-2-4B) reached a much lower share of its bandwidth ceiling, probably because of compute. That is something I was only able to know because I did the actual study. This post and the research paper that is in the works will contain even more insights that I was able to observe.

My Agent Session

The app's AI is all open-weight. For building it, I used Claude Code as a mentor and coding helper: we worked in stages, measured first, changed one thing at a time, and ran a review panel before recording the demo. That plan is in demo/plan-and-panel.md.

The Lazy Payoff

This enables one to properly document meetings that happen between people, with no manual effort other than clicking your name before you speak. In the demo, 18 seconds of speech were transcribed in 4.8 seconds, and the notes were ready 21 seconds after the meeting. The review of the generated notes is a necessary step, and as we improve the app and the harness around the model, we improve the performance as well.

This can be improved and could give better results with bigger models like Gemma 4, but this project was built on a laptop that can't handle them. I expect the performance would be better with them. I would be glad if someone reproduced it with my eval harness and shared how the performance varies.

I would appreciate any feedback or criticism on how one can do a better job, as that enables me to grow, learn and improve myself. So thank you all for that in advance.

I made this application for two purposes. One was to learn and understand how SLMs work, and how one can integrate them into an app at the deepest level, by actually doing it myself. The other was to make a tool that helps my productivity, acting as my primary note-taking app. Open innovation and local models have opened a path to a future in which one can create their own personally styled assistant. Voice-based assistants used to be sci-fi film concepts, but in the storage space needed to store that sci-fi movie, we now have a model and a harness that can actually do it.

And this is just the beginning, and I am super excited to be part of this New Age of Building.

Plus Ultra