事故升級流程應以決策表呈現，而非文字敍述

Your escalation ladder should be a decision table that a test suite can replay, not four paragraphs a tired engineer has to interpret at 03:00. That single change turns your on-call runbook from a document people read into an artifact your CI can disagree with. It also gives you something far more valuable than confidence: a failing test when the freeze rule and the first commands drift apart.

Below is the schema, the router, and the test cases I would ship first. The code is an illustrative blueprint rather than a benchmarked system, so treat it as a starting shape and adapt the field names to your own paging stack.

Why prose ladders break at exactly the wrong moment

A paragraph like "escalate to the platform team if the error rate stays high" hides at least four decisions inside one sentence. Who counts as the platform team at 03:00 on a holiday? How high is high, and measured over what window? Does the freeze window change the answer? Has anyone confirmed the freeze is actually cleared?

Ask five engineers on your rotation to answer those questions from the same page, and you will usually get three different answers. Is that a training problem, or is it a representation problem? I think it is mostly representation: prose is a lossy encoding for a decision, and on-call is where lossy encodings get expensive.

A decision table fixes this by making each row answerable. Every row names the alert, the severity band, the read-only commands to run first, the owning rotation, the escalation timer, and the freeze behaviour.

The artifact: one YAML table that owns escalation and freeze

Keep the table in the same repository as your alert rules, so a rename in one place forces a change in the other. The schema below is deliberately small; each field exists because some incident taught someone that it was missing.

# runbook/escalation.yaml — illustrative schema, adapt field names
version: 1
defaults:
  max_first_command_seconds: 30
  escalate_after_minutes: 10
routes:
  - id: api-5xx-spike
    match:
      alert: ApiHighErrorRate
      severity: [critical, warning]
    first_commands:
      - kubectl get pods -n api -o wide
      - kubectl logs -n api -l app=api --tail=200
      - aws cloudwatch get-metric-statistics --namespace AWS/ALB \
          --metric-name HTTPCode_Target_5XX_Count --period 60 --statistics Sum \
          --start-time 2026-10-19T00:00:00Z --end-time 2026-10-19T00:30:00Z
    owner: api-oncall
    escalate_after_minutes: 10
    freeze:
      applies: true
      during: [change_window, month_end_billing_freeze]
      unfreeze_requires:
        - two read receipts from api-oncall
        - page acknowledged by incident commander
    handoff_fields: [customer_impact, last_deploy_sha, error_budget_burn]

Three rules keep this table honest over time. First, first_commands must contain only read-only verbs, because the first five minutes are for gathering evidence. Second, any mutating step lives in a separate list that requires explicit clearance. Third, handoff_fields is mandatory, since an escalation without a handoff packet just moves the confusion to another rotation.

A router that refuses writes while a freeze is active

The router is the only thing anyone runs during an incident, so it stays boring and short. It resolves the row, filters the commands, and raises an error instead of guessing when the freeze evidence is incomplete.

# runbook/router.py — illustrative, not executed for this post
from __future__ import annotations

import shlex
from dataclasses import dataclass

import yaml

MUTATING_VERBS = {
    "apply", "create", "delete", "patch", "replace", "scale",
    "rollout", "drain", "cordon", "uncordon", "set", "exec",
}


class FreezeNotCleared(RuntimeError):
    """Raised when a write-class step is requested without freeze clearance."""


@dataclass(frozen=True)
class Plan:
    route_id: str
    commands: tuple[str, ...]
    owner: str
    escalate_after_minutes: int
    frozen: bool
    handoff_fields: tuple[str, ...]


def load_runbook(path: str) -> dict:
    with open(path, encoding="utf-8") as fh:
        return yaml.safe_load(fh)


def is_read_only(command: str) -> bool:
    tokens = shlex.split(command)
    if len(tokens) < 2:
        return False
    return not any(token in MUTATING_VERBS for token in tokens[1:3])


def plan_for(runbook: dict, alert: str, severity: str,
             freeze_active: bool, evidence: list[str] | None = None) -> Plan:
    route = next((r for r in runbook["routes"]
                  if r["match"]["alert"] == alert
                  and severity in r["match"]["severity"]), None)
    if route is None:
        raise LookupError(f"no route for {alert}/{severity}: page the platform owner")

    offenders = [c for c in route["first_commands"] if not is_read_only(c)]
    if offenders:
        raise FreezeNotCleared(f"{route['id']}: write verb in first_commands: {offenders}")

    freeze_rule = route.get("freeze") or {}
    frozen = bool(freeze_rule.get("applies")) and freeze_active
    if frozen:
        missing = set(freeze_rule.get("unfreeze_requires", [])) - set(evidence or [])
        if missing:
            return Plan(route["id"], tuple(route["first_commands"]), route["owner"],
                        route["escalate_after_minutes"], True,
                        tuple(route.get("handoff_fields", [])))

    return Plan(route["id"], tuple(route["first_commands"]), route["owner"],
                route["escalate_after_minutes"], frozen,
                tuple(route.get("handoff_fields", [])))

Notice what the router does with an unknown alert: it raises LookupError and names a default owner. A silent empty plan is worse than an exception, because an empty plan looks like calm.

Table-driven tests: incidents as data rows

A rehearsal only proves something if it can fail. Encode the questions your rotation argues about as parameterised rows, then let the suite answer them consistently on every pull request.

# tests/test_runbook.py — illustrative
import pytest

from runbook.router import FreezeNotCleared, is_read_only, load_runbook, plan_for

RUNBOOK = load_runbook("runbook/escalation.yaml")
EVIDENCE = ["two read receipts from api-oncall",
            "page acknowledged by incident commander"]


@pytest.mark.parametrize(
    "alert,severity,freeze_active,evidence,expected_route,expected_frozen",
    [
        ("ApiHighErrorRate", "critical", False, [], "api-5xx-spike", False),
        ("ApiHighErrorRate", "critical", True, [], "api-5xx-spike", True),
        ("ApiHighErrorRate", "critical", True, EVIDENCE, "api-5xx-spike", False),
        ("DiskPressure", "warning", False, [], "node-disk", False),
    ],
)
def test_route_and_freeze(alert, severity, freeze_active, evidence,
                          expected_route, expected_frozen):
    plan = plan_for(RUNBOOK, alert, severity, freeze_active, evidence)
    assert plan.route_id == expected_route
    assert plan.frozen is expected_frozen


def test_first_commands_are_read_only():
    for route in RUNBOOK["routes"]:
        for command in route["first_commands"]:
            assert is_read_only(command), f"{route['id']}: {command}"


def test_every_route_has_handoff_fields():
    for route in RUNBOOK["routes"]:
        assert route.get("handoff_fields"), route["id"]


def test_unknown_alert_raises_instead_of_guessing():
    with pytest.raises(LookupError):
        plan_for(RUNBOOK, "SomethingNobodyNamed", "critical", False)

Run it with python -m pytest tests/test_runbook.py -q in CI, and add one more test that fails when an alert rule exists without a matching row. That coverage check is the part most teams skip, and it is the one that catches the alert nobody documented during the last incident.

Freeze and unfreeze, written as evidence rather than judgement

The freeze rule is the field people argue about most, so make it boring. A freeze that applies during change_window returns read-only commands only, and it names the exact evidence strings that lift it.

When the evidence is incomplete, the router still returns the diagnostic commands. That is intentional: gathering evidence during a freeze is allowed, and blocking it would push people to work around the tool. Unfreeze is a separate, explicit act with two named receipts, which means the audit trail lives in the incident channel rather than in someone's memory.

One caveat worth stating plainly: this table proves consistency, not correctness. If your severity bands are wrong, the tests will faithfully protect the wrong behaviour.

Where free model access and a free server fit

MonkeyCode is an open-source coding assistant that the team offers with free model access and a free server option, and both are relevant here in narrow ways. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

• Drafting candidate rows. Paste two or three sanitised incident summaries and ask for candidate routes entries, then rewrite every command yourself and check it against your own CLI before it lands in the table.
• Generating the boring tests. The parametrised cases are mechanical, and a model is genuinely good at proposing edge rows such as severity mismatches or missing handoff fields.
• Running the rehearsal harness. The free server option is enough to run pytest and a weekly cron job that re-validates the table, without paying for a VM that sits idle.

Two cautions. Do not let a model write commands that touch production without a human reading them line by line. And do not treat free-tier availability as permanent, since the terms as of October 2026 should be re-checked on the project page before you build a dependency on them. If the limits change, the YAML and the tests move with you; only the harness host needs replacing.

Limitations and who should skip this

This approach earns its keep when you have several services, a real rotation, and at least one freeze window per month. It earns a lot less elsewhere.

• Single-service teams get little from a routing table with one row; a short checklist is cheaper.
• Teams without versioned runbooks should write the runbook first and encode it second.
• Heavily regulated environments may still require human approval for every write, in which case the router should only ever print plans.
• Anyone expecting the tests to validate reality will be disappointed; they validate agreement between your alerts, commands, and freeze policy.

Ship it this week

1. Extract every alert name from your rules into a plain list.
2. Write one YAML row per alert, with read-only commands and an owner.
3. Add the three invariant tests before you add the router.
4. Wire the suite into CI and let it fail once on purpose.

What would your ladder look like if the next argument about escalation had to be settled by a failing test instead of a loud channel?