單點故障:為何一台設備可令整個系統癱瘓,以及如何修復
Single Point of Failure: Why One Box Can Take Down Your Whole System (and How to Fix It)
單點故障(SPOF)是任何單一元件失效便會令整個系統或關鍵功能不可用的環節;其他層加入冗餘,仍無法補救請求路徑中單一依賴造成的故障。串聯的負載平衡器、應用層及資料庫示例,可用率約為99.79%;兩個各有99.9%可用率且故障互相獨立的資料庫副本,則可令資料庫層可用率達99.9999%。DNS、設定、部署、區域、憑證、第三方 API 及人員知識也可能構成單點故障。
You added three app servers. You put a load balancer in front of them. You feel good about your architecture.
Then the database server reboots for a kernel patch, and your entire product goes dark for 20 minutes.
That's a single point of failure (SPOF), and it's one of the most common reasons "highly available" systems turn out to be anything but. In this post, we'll go beyond the textbook definition and cover:
- What a SPOF really is (and why redundancy upstream doesn't save you)
- The availability math that shows exactly how much a SPOF costs you
- The hidden SPOFs most teams miss: DNS, config, certificates, people
- Concrete patterns to eliminate them, and their trade-offs
- A step-by-step method to hunt SPOFs in any design
- How to talk about SPOFs in a system design interview
Let's dive in.
What Is a Single Point of Failure?
A single point of failure is any component whose failure, on its own, makes the whole system (or a critical feature) unavailable.
The key words are on its own. If one thing breaking is enough to take you down, that thing is a SPOF.
Look at the diagram above. Three app servers look resilient. But every request eventually funnels into one database. Kill that box, and 100% of requests fail.
This is the trap: redundancy in one layer doesn't help if another layer is still singular. Availability is only as strong as the weakest layer in the request path.
A SPOF isn't always a server, either. It can be:
- A component: one database, one load balancer, one message broker
- A location: one data center, one availability zone, one region
- A dependency: one DNS provider, one payment gateway, one identity provider
- A process: one deploy pipeline, one config push that hits every server at once
- A person: the one engineer who knows how the billing system works
Why SPOFs Matter: The Math
Most articles stop at "SPOFs are bad for reliability." Let's actually put numbers on it, because this is where the intuition clicks.
Components in series multiply
When a request has to pass through several components and each one must work, you multiply their availabilities:
Load balancer (99.99%) × App tier (99.9%) × Database (99.9%)
= 0.9999 × 0.999 × 0.999
≈ 0.9979 → 99.79% available
That's roughly 18 hours of downtime per year, and it's worse than any individual component. Every SPOF you add to the chain drags the whole system down.
Redundant components compound
Now put two database copies side by side, where either one can serve traffic. The system only fails if both fail at the same time:
1 − (0.001 × 0.001) = 0.999999 → 99.9999% available
That's about 30 seconds of downtime per year for the database layer, from two components that are each "only" 99.9%.
The catch nobody mentions
That parallel formula assumes the copies fail independently. If both replicas sit in the same rack, share a power supply, run the same buggy config, or live in the same region, they'll fail together, and you're back to a single point of failure wearing a disguise.
This is why senior engineers talk about failure domains, not just copies. Two servers in one failure domain is still one failure domain.
The Obvious SPOFs
These are the ones every tutorial lists, and for good reason:
| Component | Why it's a SPOF | Typical fix |
|---|---|---|
| Database | All reads and writes depend on it | Primary + replicas, automatic failover, or a distributed DB |
| Load balancer | Every request passes through it | Pair of LBs (active-passive with a floating IP) or a managed, multi-AZ LB |
| App server | One process serving all users | Stateless servers behind an LB, auto-scaling group with N+1 capacity |
| Network link | One path to the internet | Multiple ISPs / uplinks, multiple AZs |
| Cache | If the app can't function without it, it's critical | Replicated cache cluster, and graceful fallback to the DB |
A note on caches: a cache shouldn't be a SPOF by design, but it often becomes one. If your database can't handle the traffic when the cache goes cold, then the cache is in your critical path. Plan for a "cache is down" scenario.
The Hidden SPOFs (Where Real Outages Come From)
Here's where this post goes further than most. In practice, big outages rarely come from "the database server died." They come from things nobody drew on the architecture diagram.
1. DNS. If your DNS provider (or your DNS config) breaks, nobody can find your servers, even if every one of them is healthy. In October 2021, a faulty network configuration change at Facebook made its DNS servers unreachable, and Facebook, Instagram, and WhatsApp went offline for roughly six hours. Internal tools broke too, slowing recovery.
2. Configuration and deployments. A bad config or code push that reaches every server at once turns your redundant fleet into a single point of failure. The July 2024 CrowdStrike incident is the textbook example: one faulty content update crashed millions of Windows machines worldwide at the same time. Redundancy doesn't help when every copy receives the same bug.
3. A single region. Multi-AZ protects you from a data center failure, not from a regional problem. Large regional outages at major cloud providers (including a widely felt AWS us-east-1 incident in October 2025 tied to DNS resolution for a core database service) have shown how many products quietly depend on one region, or on a "global" service that actually lives in one.
4. Certificates and secrets. If all your TLS certificates expire on the same day, or every service needs one secrets store at startup, that's a SPOF on a timer.
5. Shared infrastructure. One Kafka cluster that every service publishes to. One Redis used for sessions, rate limiting, and queues. One auth service every request has to call. Microservices don't remove SPOFs on their own; they often just move them into shared platform components.
6. Third-party APIs. Your system might be multi-region, but your payment provider, email service, or LLM API might not be. If checkout dies when one vendor dies, that vendor is your SPOF.
7. People and knowledge. If only one person can operate, debug, or deploy a system, you have a bus-factor problem. Runbooks, on-call rotation, and shared ownership are reliability work too.
Strategies to Eliminate SPOFs
1. Redundancy (N+1 and beyond)
Run more copies than you need. N+1 means you can lose one instance and still handle full load. N+2 lets you lose one during a planned deploy and survive an unplanned failure at the same time.
The important part: redundant copies need enough spare capacity. If you run two servers at 70% CPU each, losing one means the survivor needs 140%. That's not redundancy, that's a cascading failure waiting to happen.
2. Failover: Active-Passive vs. Active-Active
Redundancy only helps if traffic actually moves to the healthy copy. That's failover, and there are two main flavors:
| Active-Passive | Active-Active | |
|---|---|---|
| How it works | Primary serves traffic; standby takes over on failure | All nodes serve traffic simultaneously |
| Failover time | Seconds to minutes (detect + promote) | Near-instant (survivors already live) |
| Cost | Standby capacity sits mostly idle | All capacity is used |
| Complexity | Simpler, easier strong consistency | Harder: write conflicts, data sync |
| Good fit | Relational DB primaries, stateful services | Stateless app tiers, CDNs, multi-region reads |
Watch out for split-brain. In active-passive setups, if the network between primary and standby breaks, both might think the other is dead and both start accepting writes. The standard fix is a quorum: use an odd number of nodes (3, 5) and only allow the side with a majority to act as primary. That's why tools like etcd, ZooKeeper, and Raft-based databases run in groups of three or five.
3. Redundant Load Balancing
A single load balancer is a classic hidden SPOF. Fix it with:
- A pair of load balancers sharing a floating/virtual IP (e.g., keepalived with VRRP) in active-passive mode
- A managed cloud load balancer that is already distributed across zones
- DNS-level distribution across multiple load balancers or regions, with health checks
4. Data Replication
For stateful components, redundancy means replication:
- Synchronous replication: a write isn't confirmed until the replica has it. No data loss on failover, but higher write latency.
- Asynchronous replication: faster writes, but the replica may lag. On failover, you can lose the most recent writes.
This is the trade-off between RPO (how much data you can afford to lose) and latency. In an interview, naming that trade-off out loud is a strong signal.
5. Spread Across Failure Domains
Put copies in different racks, availability zones, and (for critical systems) regions. Here's what a basic SPOF-free setup looks like:
Every layer has at least two copies, and the copies don't share a failure domain.
6. Limit the Blast Radius of Changes
Since config and deploys are a top cause of outages, treat change as a failure source:
- Canary and staged rollouts: ship to 1%, then 10%, then everyone
- Feature flags with kill switches so you can turn off a bad change without redeploying
- Cell-based architecture: split users into independent "cells" so a bad push or noisy tenant only affects one slice
7. Graceful Degradation
Not every dependency needs to be redundant. Sometimes the right answer is to fail softly:
- Recommendations service down? Show popular items instead.
- Cache down? Serve from the DB with tighter rate limits.
- Third-party API slow? Use timeouts, retries with backoff, and circuit breakers so one slow dependency doesn't exhaust your threads and take everything else down with it.
A non-critical feature should never be able to crash a critical one.
8. Monitoring, Health Checks, and Testing Failover
Failover you haven't tested is failover you don't have.
- Health checks that check real readiness (can it reach the DB?), not just "is the process alive"
- Alerts on error rates and latency, not just CPU
- Game days and chaos testing: deliberately kill a node, an AZ, or a dependency in a controlled way and watch what happens. Netflix's Chaos Monkey made this famous for a reason.
How to Hunt for SPOFs: A 5-Step Method
Use this on any system you own, or any design you're whiteboarding.
- Trace a critical request end to end. Start from the user's device: DNS → CDN → load balancer → app → cache → DB → queues → third-party calls. Write down every hop.
- Ask "what if this disappears?" for each hop. If the answer is "the request fails and there's no alternative," mark it.
- Check for shared failure domains. Two replicas in the same AZ? Same config source? Same deploy? Same vendor? Mark those too.
- Rank by impact × likelihood. A SPOF in checkout matters more than one in an internal admin page. Fix the highest-risk ones first.
- Verify with a test. Kill the component in staging (or production, carefully). Measure how long recovery takes and whether anyone noticed.
Common Mistakes
- "We have a load balancer, so we're redundant." Not if there's only one load balancer, or one database behind it.
- Replicas without failover. A replica that needs a human to promote it at 3 a.m. gives you minutes to hours of downtime, not seconds.
- Redundancy without spare capacity. If the survivors can't absorb the load, a single failure turns into a cascade.
- Same failure domain. Two copies in one AZ, or two services reading the same config, fail together.
- Never testing failover. Backups and standbys that have never been exercised often fail exactly when you need them.
- Over-engineering. Not everything needs multi-region active-active. Match the investment to the business cost of downtime.
SPOFs in the System Design Interview
Interviewers love SPOF questions because they separate candidates who memorized a diagram from those who understand trade-offs. After you sketch your high-level design, take 60 seconds to do this out loud:
"Let me walk through the failure points. The load balancer is managed and multi-AZ. App servers are stateless, so we can lose any of them. The database is the main risk, so I'll use a primary with a synchronous replica in another AZ and automatic failover. The trade-off is slightly higher write latency, which is fine for this use case. For the cache, if Redis goes down, we fall back to the DB with rate limiting so we don't overload it."
That one paragraph shows the interviewer you understand redundancy, failover, failure domains, consistency trade-offs, and graceful degradation. It's often the difference between a "hire" at L4 and a strong signal at L5.
If you want to practice this kind of reasoning on real problems (URL shorteners, chat systems, news feeds, and more), here are a few resources from Design Gurus:
- Grokking System Design Fundamentals: covers the building blocks behind this post, including redundancy, replication, load balancing, and availability.
- Grokking the System Design Interview: walks through classic interview problems end to end, including how to identify and remove bottlenecks and SPOFs in each design.
- Grokking the Advanced System Design Interview: goes deep into how real distributed systems like Dynamo, Kafka, and Chubby handle replication, quorums, and failover.
Key Takeaways
- A SPOF is any component whose failure alone takes the system down. Your availability is only as strong as your weakest layer.
- Components in series multiply availability downward; redundant components in parallel compound it upward, if they fail independently.
- The worst SPOFs are often hidden: DNS, config pushes, a single region, shared platform services, third-party APIs, and people.
- Fix SPOFs with redundancy + spare capacity, automated failover, replication, separate failure domains, safe rollouts, and graceful degradation.
- Test your failover regularly. Untested redundancy is just hope.
Next time you look at an architecture diagram, ask one question about every box: "What happens if this one disappears?" That habit alone will make your systems, and your interview answers, a lot more resilient.
來源:DEV Community · dev.to




