跳到正文
DEV Community· Michael Tuszynski·· 3 小時前AI 評分64

11 月多起系統故障或與 AI 加速漏洞發現後的補丁壓力有關

The November Outages Weren't the Attack. They Were the Defense.

AI 導讀

Michael Tuszynski 推測,11 月 12 日至 17 日多起系統故障並非網絡攻擊,而可能是 AI 加快漏洞發現後,補丁速度超出系統安全驗證能力所致。

正文

Between November 12 and November 17, Microsoft 365 went down, the FAA grounded flights, JPMorgan Chase customers got locked out, and on November 17 the NSA's classified network went dark. Apple and Schwab had their own failures in the same stretch. The official line on each one was some version of "not a cyberattack." The NSA called its outage "scheduled routine maintenance."

Take that line at face value. It's the less comforting explanation. My hypothesis is that these outages look like the defense: production systems absorbing a patch rate they were never built for, set by AI tools that collapsed the time from bug to exploit. Nobody has published proof of that for any single incident, and I'll get to why. But if the hypothesis is right and it makes you nervous, you're reading it correctly.

An attacker can be stopped. A rate can't.

An adversary is a bounded problem. You attribute the activity, block the infrastructure, indict people, seize servers. It's slow and imperfect, but there's an end state.

What came out of the Hill briefings has no end state. Sen. Mark Warner put it bluntly: "If you're trying to patch 10 or 12 in a single week, you have these kind of outages." DNI Jay Clayton testified the incidents were unrelated but said the rate of patching is increasing. The day before, he told Fox Business the NSA outage came from patching last month's PHP flaws: "The patching that has gone on following that has caused some of these systems to not come back up when you patch them as expected." The NSA itself stuck with "routine maintenance." Both can be true. Warner's hypothetical describes how routine maintenance breaks: 10 or 12 patches crammed into a single week.

AI made bugs cheaper to find than to fix

PHP2Shell (CVE-2026-109618 and 109619) was found by Wiz Atlas working with Gemini 4 Argon and GPT-5.6 Sol, with Gemini making the key contribution, and affects every PHP release of the past decade. PHP runs roughly 70% of websites, and an EU agency warned that some sites might need to come offline while patches were prepared.

Then Nginx. This week it shipped a fix for CVE-2026-123007, a 10-year-old unauthenticated remote code execution flaw in its HTTP/2 handling. Four separate teams found it using AI tooling. Shodan counts 56.7 million Nginx hosts, and 98.9% of those that disclose a version report one in the affected range. Xint, one of the vendors involved, says its platform scanned 50,000 lines of code in 90 minutes. That's a vendor throughput claim, and lines scanned aren't bugs confirmed, so discount it.

The number that doesn't need a vendor's word: CVEs issued this year passed 100,000, more than double 2025. A CVE count isn't a deployment count. Most of those 100,000 never touch a given shop. But the ones that do arrive as vendor updates, and each update a team actually runs is a change somebody has to push into production.

The patch became the attack map

Finding bugs faster would be survivable if exploitation stayed slow. It didn't.

Amazon's MadPot honeypots saw PHP2Shell being probed on October 17, four days before the October 21 patch, a date that rests on Amazon's own telemetry alone. Google's Threat Intelligence Group documented exploit_agent, the first fully automated pipeline it has seen that reads a patch diff and generates a working exploit with no operator involved.

The disclosure norms moved in response. Project Zero cut its deadline from 90 days to 60, dropping to 30 in January 2028, and eliminated the 30-day soak period, because a published patch now hands attackers the map. Anthropic moved to 30 days for high and critical findings.

Name the incentive loop here: Google, Anthropic and OpenAI all sell the models doing the finding, and urgency is good for that business. But the four-day gap is a honeypot observation, not a forecast. Run the arithmetic for a shop on a 30-day patch SLA: exposure starts October 17 and ends around November 20. That's 34 days on a flaw affecting every PHP release in a decade. The SLA that passed audit last year is now a month of open door.

Microsoft's outage was boring, and that's the evidence

The strongest counterargument comes from Rapid7's Christiaan Beek, who says there's no public evidence that rushed patches caused any of these outages, and "complex systems fail in boring ways far more often than because of an adversary." Microsoft's preliminary report backs him up. The cause was a networking-gateway bug introduced in July that lab load tests missed, because the labs ran on weaker hardware with fewer cores than production. A Teams failover pushed traffic up while 14% of gateway capacity was already offline. Within two hours, more than 80% of gateway servers stopped responding. The FAA ground stop has been traced to a replacement router at a radio site in Arkansas that flooded a communications system with about 1,000 connection attempts every four to five seconds. No PHP in sight.

Beek is right about the per-incident causes. Look at the mechanism inside the Microsoft failure, though. A change reached production after passing verification that didn't match production: different core counts, different load shape. That's the general form of every boring outage. The FAA's failure was a hardware swap, new routers replacing units built in 2001. A change lands faster than the checks around it can tell you it's safe, or the checks measure a system that isn't the real one.

The thesis doesn't need the NSA outage to be a PHP patch. It needs the patch rate to be rising faster than the change-safety rate. The first half is documented: the CVE count doubled. The second half is my argument, and Microsoft's own report is the best support for it. Nobody's test labs doubled in fidelity this year. Pour more changes through the same verification capacity and the boring failures arrive more often, for everyone, in the same weeks.

There's no safe setting on the dial

Patch slowly and you get PHP2Shell, exploited before a fix existed. Patch quickly and you get the NSA's week. Turning the dial picks which failure you want.

The only exit is making the patch itself safe at speed. That means four things: rollout ordered by blast radius, test environments that match production hardware and traffic mix, rollback that's been rehearsed rather than documented, and proof the fix is running.

The last one gets skipped most. Miggo's CEO, talking to CSO about the Nginx fix, laid out the problems: version numbers lie because distros backport fixes without changing them, "a reload doesn't swap out the running code," and the attack rides HTTP/2 traffic that WAFs and access logs never see. You can't confirm this patch from the outside, and you can't trust the version string from the inside.

Proving the fix is running means checking the process

This failure mode isn't unique to Nginx. A local inference server in my lab kept answering its version endpoint for two days while every embedding call returned HTTP 500. The watchdog stayed green the whole time and the indexer wrote zero chunks. I now probe the capability, not the port: for an embedder, embed.

During a zero-downtime reload, a health-check 200 can be served by the old process still handling traffic. I gate 'it's live' on a signal only the new build emits: a boot timestamp after the deploy or a bumped version ID. A config reload printed the new mount options while the live NFS mount kept the old ones. I verify with what is actually mounted, never by re-reading the config that says what would be mounted.

For Nginx on a Linux host, that looks like this:

#!/usr/bin/env bash
# Is the patched nginx binary the code actually running?
set -euo pipefail
BIN=/usr/sbin/nginx
MASTER=$(cat /run/nginx.pid)

# ctime, not mtime: package managers preserve mtime from the archive
INSTALLED=$(stat -c %Z "$BIN")
STARTED=$(date -d "$(ps -o lstart= -p "$MASTER")" +%s)

for pid in "$MASTER" $(pgrep -P "$MASTER"); do
  if readlink "/proc/$pid/exe" | grep -q '(deleted)'; then
    echo "FAIL: pid $pid is running a binary that's no longer on disk"; exit 1
  fi
done
(( STARTED > INSTALLED )) || { echo "FAIL: master predates patched binary"; exit 1; }

# Version strings lie under backports; check the changelog for the CVE itself
apt changelog nginx 2>/dev/null | grep -q 'CVE-2026-123007' \
  || rpm -q --changelog nginx 2>/dev/null | grep -q 'CVE-2026-123007' \
  || { echo "FAIL: installed package doesn't claim the fix"; exit 1; }

echo "PASS: running master started $((STARTED - INSTALLED))s after install"

The (deleted) check catches the reload trap Miggo described. A nginx -s reload re-reads config, but the master keeps executing the old binary image, and the kernel flags it once the package manager replaces the file. Only a full restart or a binary upgrade clears it. After any restart or deploy I re-observe and record before and after state (PIDs, listeners, HTTP status) before calling it done.

Where the squeeze doesn't land, and what the fix costs

Shops that already ship through progressive delivery with real canaries and automated rollback won't feel this the same way. For them, a doubled CVE count is a bigger queue through a pipeline built for queues. The pain concentrates on systems that grew by accretion rather than design, and on fleets where patching still means a maintenance window and a change board.

The advice isn't cheap. A production-fidelity test lab means buying production-class hardware that does nothing most of the day. Microsoft's report says its labs used weaker hardware; it doesn't say why. My inference is cost, because matching production is expensive, and that kind of savings is exactly what lets a bug like July's through. Canaries cost hours you may not have. With an unauthenticated RCE across most of 56.7 million Nginx hosts and an agent that turns patch diffs into exploits, the canary might shrink to one hour on one region. In that case rehearsed rollback is the thing you're paying for, because you'll ship before you're sure.

If my hypothesis holds, the NSA's description was accurate. Scheduled routine maintenance, applied at a rate nobody scheduled for, is how a defense takes down the thing it's defending. And the next test of that rate arrived the same week the NSA went dark: an Nginx patch for a 10-year-old hole, with 56.7 million hosts in line.

來源:DEV Community · dev.to