8 宗重大軟件故障圖解及其帶來的測試啟示

What major incident investigations teach us about inputs, interfaces, deployment, performance, timing, and recovery.

A successful test tells us something specific: under the conditions we exercised, the system satisfied the assertions we checked. Major failures often reveal the boundary of that evidence.

The eight incidents below offer concrete ways to strengthen it. Their published investigations document technical defects alongside gaps in validation, release processes, design, and operations. The proposed tests are an engineering interpretation of those findings, not a claim that a QA team alone could guarantee prevention.

Each illustration separates the failure mechanism from controls that could detect the defect or reduce its impact.

1. CrowdStrike: exercise every field

A mismatch between the content template and the real sensor inputs escaped tests that used wildcard matching.

On 19 July 2024, a CrowdStrike content update exposed a mismatch: a template expected 21 inputs, while the sensor supplied 20. Earlier wildcard tests had not required reading the missing input. An explicit match did, causing an out-of-bounds read. Microsoft estimated that 8.5 million Windows devices were affected. Sources: CrowdStrike technical RCA; Microsoft impact estimate.

A useful test would exercise every field with both wildcard and explicit criteria against the actual sensor. A separate contract check would compare declared fields with supplied values. Staged deployment and crash monitoring could then limit the reach of a defect that still escaped.

QA lesson: Configuration and content changes need executable validation against the system that consumes them.

2. Knight Capital: test the deployment

Seven servers received the new release; one stale server activated defective legacy code when a flag was reused.

On 1 August 2012, a deployment reached seven of Knight Capital's eight servers. On the remaining server, a repurposed flag activated defective legacy Power Peg code. Runaway orders produced losses exceeding $460 million in roughly 45 minutes. The SEC identified deployment and risk-control deficiencies. Sources: SEC investigation.

The release test should include an intentionally incomplete rollout. Send an order through the mixed cluster and verify that execution accounting stops further routing appropriately. Deployment checks should establish which version every server runs, while independent tests should prove that financial limits can actually stop the flow.

QA lesson: A passing build is only one part of a verified production deployment.

3. Cloudflare: make performance an assertion

The WAF rule passed functional checks, but excessive regular-expression backtracking exhausted CPU.

Cloudflare's 2 July 2019 outage lasted 27 minutes. A WAF rule caused excessive regular-expression backtracking. Existing tests checked blocking and allowing requests without detecting runaway CPU use. A CPU safeguard had also been removed, and the rule could reach the global network without a staged rollout. Sources: Cloudflare postmortem.

Performance tests need deliberately difficult inputs and explicit resource budgets. The safeguard itself deserves regression coverage: a pathological rule should trigger protection without exhausting the service. A small initial rollout can expose production effects before expansion.

QA lesson: Correct output must arrive within an acceptable resource budget.

4. Ariane 5: retest inherited assumptions

Software reused from Ariane 4 encountered values outside its assumptions under Ariane 5's flight conditions.

Ariane 5 Flight 501 was lost on 4 June 1996. Reused inertial-reference software converted a 64-bit floating-point value into a 16-bit signed integer. The new flight conditions exceeded the assumed range. Both reference units encountered the same defect, and testing had not adequately represented the new trajectory. Sources: Ariane 501 Inquiry Board report; ESA inquiry summary.

Reusing a component should trigger an assumption review: input ranges, timing, operating modes, and failure handling. Representative trajectory replay would exercise those assumptions together. Tests should also examine failures that identical primary and backup software can share.

QA lesson: Previous success supplies evidence for the previous environment. New operating conditions require new evidence.

5. Mars Climate Orbiter: verify the meaning of data

The interface required metric units, but ground software delivered impulse values in pound-force seconds.

Mars Climate Orbiter was lost in September 1999. Ground software delivered thruster-impulse values in pound-force seconds; navigation software expected newton-seconds. The interface specification already required metric units. NASA found inadequate verification of the relevant ground-system interfaces. Sources: NASA investigation, including the Phase I report.

A known-answer test can follow a physical scenario through the producer, the file, and the consumer, comparing the result with an independent calculation. For example, one pound-force second is approximately 4.448 newton-seconds. Matching field names and numeric types alone would not establish agreement.

QA lesson: An interface contract includes units, interpretation, and physical meaning.

6. Therac-25: test sequences and timing

This diagram focuses on the Tyler race condition, one of several defects documented in the Therac-25 investigations.

Therac-25 overdoses caused deaths and serious injuries during 1985–1987. The illustration focuses on the Tyler accidents: rapid edits interacted incorrectly with concurrent machine-setup tasks. The investigation also documented other defects and serious software QA weaknesses. Sources: Leveson and Turner investigation, Part III.

Testing should vary the timing of edits relative to setup, exploring different task execution orders. The central assertion is a safety property: an unsafe physical configuration must never permit beam activation. Independent interlocks also need verification, including their behavior when software reports inconsistent state.

QA lesson: The same inputs can produce different outcomes when their order and timing change.

7. Patriot: treat uptime as an input

Numerical error in time conversion increased with continuous operating time and displaced the predicted search area.

On 25 February 1991, a Patriot system at Dhahran failed to track and intercept a missile that killed 28 Americans. Limited precision in time conversion produced an error that grew with uptime. The system had been operating continuously for more than 100 hours. Sources: GAO report IMTEC-92-26.

A useful test matrix crosses target conditions with elapsed operating time. Simulated time can explore long durations efficiently, while real long-running tests check integration behavior. A higher-precision reference can reveal numerical divergence before it becomes operationally significant.

QA lesson: Duration belongs beside values, states, and sequences in the test input space.

8. NATS: test the failures a backup can repeat

The same unusual flight-plan data caused primary and backup systems to enter maintenance mode to preserve safety.

On 28 August 2023, a valid-format flight plan containing a rare six-attribute combination disrupted NATS processing. Primary and backup systems applied the same logic and entered maintenance mode to preserve safety. NATS identified omitted intended logic, while judging the original testing reasonable. Sources: NATS final investigation.

Tests can challenge identifier ambiguity and trace processing requirements to executable checks. Resilience exercises should also reproduce a failure in both systems and measure safe recovery and manual-processing capacity. The report does not justify certainty that a tester should have predicted the exact combination.

QA lesson: Redundancy needs validation against shared software failures, as well as individual hardware failures.

Turn the cases into test-plan questions

For your next consequential change, ask:

• Which fields, boundaries, and optional branches have never been exercised?
• What happens when deployment is incomplete or versions differ?
• Which resource limits are asserted under difficult inputs?
• Which assumptions came from a different environment or older product?
• Who independently verifies the meaning of data crossing an interface?
• What changes when events arrive faster, slower, or in another order?
• What changes after hours or days of operation?
• Which failure could the primary and backup systems repeat together?

Give each relevant question an observable outcome: a test, a release gate, a measurable limit, or a recovery exercise. That makes the lesson useful before the next incident.

Illustrations generated with AI and reviewed for wording and causal structure. Historical findings are linked above; the QA controls are proposed engineering lessons.