跳到正文
DEV Community· CouncilAI Dev·· 1 天前AI 評分48

CouncilAI:將 LLM council 做成 Windows 應用程式的開發心得

I built an LLM council as a Windows app. Here's what I learned

AI 導讀

一名獨立開發者將多個模型協作回答問題的 LLM council 做成 Windows 桌面應用程式 CouncilAI,由裁判模型整合各模型的回答。2.1 版加入 Agent 模式,可處理指定資料夾內的檔案,並由另一個模型檢查修改內容。CouncilAI 可在 Microsoft Store 下載,支援 Windows 10 和 11,用戶需自備 API keys。

正文

CouncilAI Council Mode with three AI models answering a question about SQL vs NoSQL
Every time I asked one AI model a question that mattered, I ended up pasting the same question into two or three others to see if they agreed. Usually they mostly did. Sometimes one of them was confidently wrong, and the only way I found out was that the others disagreed.

So I built an app that does the pasting for me. It's called CouncilAI, a Windows desktop app where Claude, GPT, Grok and any OpenRouter model answer the same question, and a judge model combines their answers into one. I'm a solo developer, it's on the Microsoft Store, and a few strangers (the first one in Brazil, a later one in India) have paid for it, which still feels a bit unreal.

This post covers how it works under the hood and a few lessons I learned the hard way.

The council idea

If you've seen Andrej Karpathy's llm-council experiment, it's the same basic idea: several models answer, then one model reads all the answers and writes the final one. His version is a local web app you set up with Python, uv and npm. I wanted something people could install from the Store and use in two minutes, with no terminal.

In CouncilAI, a council round works like this:

  1. The question goes to every model you've switched on, in parallel. Each answer shows up in its own card the moment that model finishes, instead of everything appearing at once at the end.
  2. A judge model (your pick in Settings) reads all the answers and writes one combined reply, including where the models disagreed.
  3. Follow-up questions carry the conversation history to both the members and the judge, so "now make it cheaper" still knows what "it" is.

There are also three other modes: normal single-model chat, Compare (two models side by side), and Sequential, where one model drafts, a second critiques, and a third refines.

Stack

  • Python + PySide6 (Qt) for the desktop UI
  • Provider clients for Anthropic, OpenAI, xAI and OpenRouter, all using the user's own API keys
  • Keys stored in Windows Credential Manager
  • Packaged as an MSIX and shipped through the Microsoft Store, which also handles updates

There's no backend. Every request goes straight from the user's PC to the provider they chose.

Agent mode, with a second opinion

The newest feature (2.1) is agent mode. You pick a working folder, describe a goal, and the model works through your files on its own using five tools: list_files, read_file, search_files, edit_file and write_file.

Two design choices I'm happy with:

Edits are staged, never written. Every edit_file or write_file call changes an in-memory copy. When the agent finishes, you get a diff for every file and nothing touches the disk until you tick the files and click Apply. Changed files are backed up, and the last set of changes can be undone in one click. The agent can't run commands or reach anything outside the folder, and it skips files that look like secrets (.pem, .key, .npmrc, SSH keys and so on).

A second model reviews the work. Before you see the proposal, a different model (from another provider when you have one) checks the changes against your request. If it finds real problems, the changes go back to the agent for one round of fixes. It's the council idea applied to code: one model's mistakes are easier for a different model to spot.

Tool calling was the fiddliest part. Anthropic uses its own tool-use format and the others use the OpenAI-style one, so there's a small translation layer so the same agent loop runs on any provider.

Lessons learned

1. A score that looks authoritative but isn't is worse than no score. Early versions showed a "quality score" on each council answer, based on text statistics like length and keyword overlap. In real testing every frontier model got about the same score, whether it was right or confidently wrong. I removed it completely. Now the judge model reads the answers and explains the agreement and disagreement in words, which is far more useful.

2. Handle replies that get cut off mid tool call. If a model hits its output limit halfway through writing a big edit_file call, you get half a tool call. Running it would corrupt the file. Now the app throws away tool calls from a truncated reply and asks the model to redo it in smaller changes.

3. Don't save model IDs as exact strings. I shipped a bug where a model would quietly drop out of the council. The settings saved the exact model IDs you'd ticked, and when a provider's default model changed names, the saved ID no longer matched anything. The fix was to save what you switched off instead of what you switched on, and to match direct providers by provider rather than by exact model name. New models now join the council by default.

4. Emoji are not icons. The first versions used emoji all over the UI. They render differently on every Windows build and ignore your theme colours. In 2.1 I replaced them with small line icons drawn in the theme's colours, and the app instantly looked more professional.

5. Pre-connect for a faster first reply. The first message of a session was always the slowest, because of DNS, TLS and connection setup. Now the app opens a connection to each provider in the background at startup (no key or tokens are sent), so the first real request reuses a warm connection.

Try it

CouncilAI is on the Microsoft Store for Windows 10 and 11. You bring your own API keys. If you just want to try it cheaply, OpenRouter has free models and the setup guide takes a couple of minutes. There's more at councilai.ca.

I'd love feedback, especially from people who already use several models side by side. What would make a council more useful to you?

來源:DEV Community · dev.to