別等狀態頁：如何用不足 70 行 Python 建立多窗口錯誤預算消耗率告警

• Book: System Design Pocket Guide: Fundamentals
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

Between 20 and 27 October 2026, OpenAI's status page logged authentication errors for some users near Illinois until a configuration change was rolled back, elevated error rates for gpt-5.6-luna API requests on Chat Completions and Responses, and a degraded-performance incident on 6.1 Sol, three of 44 October incidents across its products. Each ran between 60 and 78 minutes; the luna one took 70 minutes from Identified to Resolved.

The luna incident page also carries this sentence: "Availability metrics are reported at an aggregate level across all tiers, models and error types."

A status page answers "is the provider broadly up?" You need the answer to "did my users get errors in the last ten minutes?" Those are different questions, and only your own request log can answer the second one.

Two true numbers, neither of them yours

The aggregator llmapi.ai's OpenAI page shows two figures side by side, both drawn from OpenAI's own status data. One is 99.97% uptime over 90 days, labelled "OpenAI's figure for its API group." The other is 31 h 28 min in the last 30 days with at least one of 14 API components affected, across 12 incidents. The site's FAQ is explicit that "Uptime is the provider's own figure."

Both numbers are honest. They measure different things. If your product calls one model through one endpoint, neither tells you how many of your requests failed between 15:34 and 16:44 UTC on 21 October.

So stop waiting for someone else's dashboard to turn amber. Define an SLO for your own service, measure it from your own logs, and page on how fast you are spending the error budget. The method below is the multi-window, multi-burn-rate alert from chapter 5 of Google's Site Reliability Workbook, "Alerting on SLOs".

Burn rate, in one section

Pick an SLO. Say 99.9% of requests succeed over 30 days. That leaves an error budget of 0.1% of requests.

Burn rate is the observed error ratio divided by that budget ratio. The workbook defines it as "how fast, relative to the SLO, the service consumes the error budget." A burn rate of 1 spends the budget at a pace that "leaves you with exactly 0 budget at the end of the SLO's time window." A burn rate of 50 means 5% of requests are failing, and at that pace the month's budget is gone in under 15 hours.

A burn rate held for a while turns into a share of the budget. Thirty days is 720 hours, so a burn rate of 14.4 for one hour spends 14.4 / 720 = 2% of the budget. The workbook's recommended parameters for a 99.9% SLO build on that:

The alert fires only when the burn rate is over the threshold in both the long window and the short window. The long window proves the burn is big enough to matter. The short window proves it is still happening. The workbook's guideline: "make the short window 1/12 the duration of the long window."

The long window says the burn matters; the short window says it is still going on.

The script

This replays 12 hours of synthetic traffic, bucketed per minute, at 2,000 requests a minute. The baseline is one error a minute (burn rate 0.5). At minute 400 there is a one-minute spike where 30% of requests fail. At minutes 480 to 509 there is a 30-minute blip at 5% errors, burn rate 50. That is shorter than the provider incidents above, with a made-up error rate.

Four rules run side by side. Two are the workbook's page pairs. Two are single-window rules, there only so you can see what the second window buys you.

"""Multi-window, multi-burn-rate alerts over a request log."""

SLO = 0.999
BUDGET = 1 - SLO  # fraction of requests allowed to fail

# (name, long window, short window, burn-rate threshold)
# windows are in minutes
RULES = [
    ("naive-5m", 5, 5, 14.4),     # one window, for contrast
    ("1h-only", 60, 60, 14.4),    # one window, for contrast
    ("page-fast", 60, 5, 14.4),   # SRE workbook pair 1
    ("page-slow", 360, 30, 6.0),  # SRE workbook pair 2
]

def synthetic_log(minutes=720, per_min=2000):
    """Per-minute (requests, errors) buckets.

    Baseline: 1 error a minute (0.05%, burn rate 0.5).
    Minute 400: a one-minute spike, 600 errors (30%).
    Minutes 480-509: a 30-minute blip, 100 errors (5%).
    """
    log = []
    for m in range(minutes):
        errors = 1
        if m == 400:
            errors = 600
        elif 480 <= m < 510:
            errors = 100
        log.append((per_min, errors))
    return log

def prefix_sums(log):
    reqs, errs = [0], [0]
    for r, e in log:
        reqs.append(reqs[-1] + r)
        errs.append(errs[-1] + e)
    return reqs, errs

def burn_rate(reqs, errs, end, width):
    """Burn rate over the `width` minutes ending at `end`."""
    start = max(0, end - width)
    r = reqs[end] - reqs[start]
    e = errs[end] - errs[start]
    return (e / r) / BUDGET if r else 0.0

def replay(log):
    reqs, errs = prefix_sums(log)
    firing = {name: False for name, *_ in RULES}
    for minute in range(len(log)):
        end = minute + 1
        for name, long_w, short_w, threshold in RULES:
            on = (
                burn_rate(reqs, errs, end, long_w) > threshold
                and burn_rate(reqs, errs, end, short_w) > threshold
            )
            if on != firing[name]:
                state = "FIRING" if on else "resolved"
                print(f"minute {minute:>3}  {name:<10} {state}")
                firing[name] = on

if __name__ == "__main__":
    replay(synthetic_log())

No dependencies, standard library only. Prefix sums make every window lookup two subtractions, so the replay stays fast even if you feed it a month of minutes.

What fires, and what stays quiet

The log uses round numbers on purpose, so you can work out every transition by hand. Here is what the arithmetic says each rule does. I haven't pasted a terminal run here, so save the file and check the minutes against your own run.

The spike and the blip get very different treatment.

The spike never pages anyone through the workbook rules. The naive five-minute rule wakes someone up at minute 400 for an event that cost 599 extra errors. Against a 30-day budget of 86,400 failed requests (0.1% of 2,000 × 43,200 minutes), that is under 1%. The page-fast pair stays quiet because the one-hour window only reaches a burn rate of about 5.5.

The blip pages at minute 496, 16 minutes after it starts. By then 17 blip minutes have cost 1,700 errors, almost exactly 2% of the budget. That is the 14.4 threshold doing what the table says it does. The page-slow pair never fires on this log: its six-hour window tops out at 3,929 errors, short of the 4,320 it needs. A 30-minute incident is the fast pair's job. A slow, steady burn over several hours is the slow pair's.

The short window fixes reset time. The blip's last bad minute is 509. page-fast resolves at 513. The 1h-only rule keeps paging until minute 553, 44 minutes after the errors stopped, because a one-hour average remembers the blip. The workbook makes the same point: the multiwindow alert shows "a better reset time by ceasing to fire five minutes later, rather than one hour later."

The aggregate can look calm while your one endpoint is burning budget.

Wiring it to a real request log

The replay takes per-minute (requests, errors) buckets. Building them from a log of your outbound calls to the provider takes about ten lines:

def bucket(lines, start_ts):
    """(unix_ts, status) pairs -> per-minute buckets."""
    out = {}
    for ts, status in lines:
        m = int(ts - start_ts) // 60
        r, e = out.get(m, (0, 0))
        # status 0 = client-side timeout
        bad = status == 0 or status >= 500 or status == 429
        out[m] = (r + 1, e + (1 if bad else 0))
    last = max(out, default=-1)
    return [out.get(m, (0, 0)) for m in range(last + 1)]

Decide what "bad" means before you decide thresholds. A timeout your client gave up on is a failure your user saw, even though the provider never returned a status, so record it. A 429 counts too: whether it was the provider's capacity or your own quota, the user got an error. If your product has a latency promise, add slow responses as a second SLO rather than mixing them into this one.

Measure in two places. An SLO on what your users get from your service is the one that pages. An SLO on your outbound calls to each provider tells whoever is paged where to look, whether or not a status page has an entry for it yet.

For a real deployment, add the ticket tier with ("ticket", 4320, 360, 1.0) once you keep three days of buckets. And if you already run Prometheus, you probably don't want to keep this script running. The workbook gives the same logic as a Prometheus expression, and tools generate the rules from an SLO spec: Sloth autogenerates multi-window, multi-burn-rate page and warning rules for Prometheus, and Pyrra generates four multi-burn-rate alerts at different severities. The script is still useful as a way to see exactly what those rules will do before you trust them with your pager.

One caveat from the workbook itself: at low traffic, a single failed request is a large share of a minute's requests, and "it's harder to automatically distinguish unimportant events in low-traffic services." The chapter has a section on low-traffic services. Read it before you point these thresholds at an endpoint that sees ten requests an hour.

If this was useful

Burn-rate alerts tell you a dependency is hurting your users. What happens next is a design question: what your service does while the dependency has a bad hour. System Design Pocket Guide: Fundamentals covers the building blocks you reach for when that page fires, from caching and rate limits to load balancing, latency and availability.