Ivory 新增 etcd 支援，提供叢集監察及主節點切換

See the leader, the Raft lag and the alarms of every etcd member in one screen — and move the leader with one click

etcd is one of those databases you don't think about. It sits quietly under your Patroni cluster, or under your Kubernetes control plane, and keeps everyone agreeing on who the leader is.

Until it doesn't. When etcd hurts, everything built on top of it hurts too. Patroni can't renew its leader lock, the Postgres primary demotes itself, and suddenly your "database incident" is actually an etcd incident.

I added etcd support to Ivory for exactly this reason. Patroni uses etcd as its coordination store, so if Ivory looks after Patroni, it should look after etcd too.

The problems

Here's what troubleshooting etcd usually looks like for me:

etcdctl needs a lot of flags. Endpoints, certificates, --write-out=table, --cluster… Every time, I have to remember the exact incantation, and type it on the right host.
"Who is the leader?" is a three-endpoint question. etcdctl endpoint status against every member, then reading a wide table to find the one that says true.
Alarms are silent. When etcd runs out of space it raises a NOSPACE alarm and stops accepting writes. Nobody tells you unless you ask with alarm list.
A member from another cluster looks perfectly healthy. If you add the wrong address, it responds, it has a leader, and everything looks fine — it's just not your cluster.
Lag is hidden in Raft indexes. A slow member falls behind on applied entries, and you only notice when requests start failing.

How Ivory solves them

One overview for the whole cluster

Add your etcd cluster to Ivory by typing the node addresses, or just give it one node and let Ivory detect the rest — etcd's member list is the same from whichever node you ask.

The overview shows every member at once:

Role — the member etcd names as leader is the leader, the rest are replicas.
State — whether the member is actually running, observed, not assumed.
Lag — how many Raft entries a member is behind the leader. Ivory warns past 1000 entries.
Tags — the etcd version, the Raft term, and whether a member is a learner.

Each member keeps its etcd name, so what you see in Ivory is exactly what etcd calls it.

Warnings that actually tell you something

Ivory shows two kinds of warnings, labelled separately so you always know who's talking:

What etcd itself reports — alarms like NOSPACE, a lost leader, and a backlog of committed but not yet applied entries big enough for etcd to start refusing requests.
What Ivory notices — a member it can't reach (naming the address it tried, so a routing problem reads differently from a dead member), a node that isn't in the cluster's member list, or a lag past the threshold.

And because every node reports its etcd cluster id, a member from a different cluster is flagged right away instead of rendering as a happy, healthy node.

Move the leader

etcd elects its leader by itself, so there's no failover or restart to manage. But sometimes you want the leader somewhere else — before maintenance on its host, for example. Ivory offers a switchover: pick the member that should lead, and it moves leadership there.

A console without the flags

Every node has an etcdctl-style console: get (with --prefix, --limit, --keys-only), put, del, member list, endpoint status, endpoint health and alarm list. No endpoints, no certificates to type — Ivory already knows them.

There are also saved queries for the things you check most: the member list, endpoint status and health, alarms — and the keys Patroni stores for its clusters. That last one is a lifesaver when you need to know which Postgres node Patroni thinks is the leader.

The machine behind each node

Give Ivory an SSH key, and each node also shows the machine it runs on: CPU, memory, network, the process list and any log file on the host. If etcd runs in Docker, the Container tab shows its logs, lets you inspect, start, stop or restart it. Most etcd problems are disk or network problems, and this is where you see them — no extra SSH session needed.

Need a new etcd cluster?

Ivory can deploy one for you over plain SSH, from a template:

Etcd (Multi Host) — three nodes etcd1–etcd3, one per machine, client port 2379 and peer port 2380. Replace the example addresses in the member list with your hosts.
Etcd (Single Host) — all three on one machine on client ports 2479, 2481 and 2483. Perfect for a test environment.

Both leave authentication off on purpose: that's what lets Ivory's Patroni templates use the cluster as their coordination store right away. Deploy etcd first, then Patroni on top of it, and you have a full high-availability Postgres setup in a few clicks.

Ask your AI agent

Ivory has a built-in MCP server, so you can also connect Claude, Cursor or Codex and simply ask:

"Is my etcd cluster healthy? Any alarms?"

"Which etcd member is the leader, and is any member behind?"

"What does Patroni store in etcd for the prod cluster?"

"Move the etcd leader to etcd2 — check that it's caught up first."

The agent works through Ivory, with exactly the permissions your user has. Details are in the MCP docs.

Try it

docker run -p 80:80 --restart always veegres/ivory

Open http://localhost, go through the setup wizard, and add your etcd cluster. etcd support is currently in beta — it works end to end, but it has had less time in the field than Postgres, so feedback and issues are very welcome.

Keep your Raft afloat, and it will keep everything above it standing. 😉

Useful links: