黑色星期二(1929 年 10 月 29 日):從級聯崩潰汲取的 4 個工程教訓
Black Tuesday · Oct 29, 1929 — 4 engineering lessons from a cascade crash
1929 年黑色星期二的股災顯示,槓桿、連鎖賣出規則及資訊延誤可以疊加,引發級聯失效。文章從分散式系統故障角度提出 4 個工程教訓:防止級聯、控制槓桿效應、重視觀測能力,以及透過壓力測試和故障演練做好準備。
Ninety-six years ago today, the New York Stock Exchange lived through what we now call Black Tuesday.
It wasn’t a single overnight break. The week before had already fallen hard; Black Monday (Oct 28) kept sliding; on the 29th, selling hit like an avalanche. By 1932 the Dow had lost about 89% from its peak. Panic gets the headline, but leverage, chained sell rules, and slow information stacked into a cascade failure.
Read it as a distributed-systems outage and you get four takeaways:
Cascades beat single points
One node can fail; everyone firing the same sell rule at once takes the whole mesh down. Circuit breakers, rate limits, and bulkheads exist to stop stampedes.Leverage turns a wobble into a disaster
Borrowed money, borrowed size, borrowed confidence — pretty on the way up, lethal on the way down. Retry storms, unbounded queues, and call chains with no backpressure are tech’s version of leverage.Slow information makes bad decisions
Quotes moved by slips and phones; a half-beat late meant a wrong call. Laggy metrics and fuzzy alerts do the same to on-call engineers. Observability is a brake light, not decoration.“Looks fine” is often the night before the storm
Healthy systems pressure-test and run failure drills — they don’t wait for their own Black Tuesday to teach the lesson.
(Some images are AI-generated.)
What’s the cascade failure you remember most? Which link broke first?
來源:DEV Community · dev.to