跳到正文
DEV Community· SAMUELFMEDEIROS·· 3 小時前AI 評分55

監控守護機制若從不觸發,便等同失效

A guard that never fires is the same as a guard that does not work

AI 導讀

文章指出,守護檢查必須強制觸發它原本要捕捉的條件,否則持續顯示「OK」並不能證明它正常運作。文中以 `pgrep -f my-service` 匹配到 watchdog 自身、以及 `active_chats=0` 可能源於讀錯資料庫為例,提醒檢查程序辨識方式與資料來源。作者建議逐次記錄測量值和判斷結果,並以已送達訊息的 ID 核實告警,而非只看腳本退出碼。

正文

Every guard has two states: working and green by accident. Both print “OK”. The
difference only shows when you force the condition it was supposed to catch.

The counter that counts itself

pgrep -f my-service inside the watchdog itself matches the shell running the
command
. The guard sees 1 process, concludes “alive”, and never restarts anything — including
when the service is dead.

pgrep -f '[m]y-service' # the bracket prevents the self-match
Better: scan /proc/<pid>/cmdline excluding $$ and
$PPID, and confirm with a second signal (log age, service state).

Zero is not health

A session guard reporting active_chats=0 is either saying “nothing to do” — or
“I am reading the wrong database”. If three groups are talking and the guard counts zero, that
number is a symptom, not peace of mind.

Before trusting a zero, confirm the source: is the file the guard reads the same one the process
writes? A store with 0 rows keeps the guard green forever.

A fixed gate on an oscillating machine

A fixed memory floor (say 4,600 MB) on a machine that measures 10,800 MB and 4,100 MB minutes
apart blocks half the runs — and you blame the service.

Size the gate by the minimum observed with explicit headroom (one task = 400-800 MB → a
2,000 MB floor gives 2.5x). Measure twice, minutes apart, before picking the number.

A lost delivery looping

Dedup by text hash resends the same message every time the text changes — and the text
changes each tick, because the agent keeps working. Result: the group gets the same notification
several times.

A lost delivery is one event: dedup by (channel, ready_timestamp), and store
the key even when the resend fails — otherwise it repeats forever.

How to prove a guard works

  • Negative test, mandatoryForce the condition (kill the service, empty the source,
    delay the delivery) and verify the guard fires. A guard that never failed a forced test has
    no evidence of working.

  • Log a verdict per tickWithout a log line per run, “green” is faith. The log must
    say what was measured and what was decided.

  • Delivery proofAn alert that does not arrive is an alert that does not exist:
    demand the delivered message id, not the script's exit code.

See the Roger MLOps Guard skill
See all skills

Want this wired into your project? Guided setup from
R$ 1.500, with the first cycle run together.

← all skills

來源:DEV Community · dev.to