Google Cloud 雙平面可觀測性架構：信號契約、告警政策與六角形分派器

We stopped learning about production failures from other teams by splitting observability into two planes: telemetry that's always on, and notification that's deliberate and rare. This is how we built it: the contracts, the config and the sharp edges we hit along the way.

Our services happen to be written in Go, so the code samples are Go. The architecture isn't: every boundary is OpenTelemetry, Cloud Monitoring or JSON on Pub/Sub, so a workload in any language can plug in.

Context in 60 seconds

• The fleet: a GKE cluster of Go services plus more than a dozen Cloud Run jobs, across three environments (dev, preprod, prod), each in its own GCP project.
• Before: every component instrumented differently, logs as the only real signal, and one Teams channel per service.
• The trigger: a sub-step in a Cloud Run job failed; the job swallowed the error and exited 0. No alert fired. A downstream team noticed hours later.
• The goal: any failure that needs a human reaches one within minutes, through one predictable channel. Everything else stays quiet, but visible.

Architecture at a glance

What's language-specific, and what isn't

The only code that has to exist per language is the instrumentation library. Everything downstream depends on what lands in Google Cloud, not on how it was produced:

A Python, Java or Node.js service joins the platform by emitting the same signals and publishing the same JSON. The policies, dashboards and dispatcher don't change.

Two planes that never mix

The telemetry plane is always on. Every workload emits the same signals, all the time, straight to Google Cloud Observability. Dashboards and Error Reporting read Cloud Monitoring and Cloud Logging directly. Nothing in this path goes through a service of ours.

The notification plane is deliberate and rare. Nothing reaches a person unless someone decided, on purpose, that it's worth interrupting someone for. Most of the time that decision is a Cloud Monitoring alert policy on the shared signals. Occasionally it lives in code, for app-level events that can't be expressed as a metric.

Pub/Sub carries alerts, never telemetry. It's tempting to send metrics through the same bus, but then a service of ours would have to aggregate time series, store history and evaluate alert conditions, all things Cloud Monitoring already does. It would also sit in the path of every signal: if it went down, we'd lose visibility exactly when we needed it.

One instrumentation library per language

If every team instruments differently, you can't build anything on top. So each language gets exactly one small library, and it's the only sanctioned way to wire up telemetry. Our fleet is Go, so we have one. This is what using it looks like in a Cloud Run job:

ocfg := obs.Config{
    GCPProject:     cfg.Project, // empty = telemetry off (local runs)
    ServiceName:    "report-syncer",
    ServiceVersion: buildSHA,
    Environment:    cfg.Env, // dev | preprod | prod
    ComponentType:  obs.ComponentJob, // or obs.ComponentService
}
shutdown, err := obs.InitOTel(ctx, ocfg)
if err != nil {
    return fmt.Errorf("init otel: %w", err)
}
defer func() {
    // Fresh context: the caller's ctx may already be canceled,
    // and this flush carries the run's final result.
    ctx, cancel := context.WithTimeout(
        context.Background(), 5*time.Second)
    defer cancel()
    if err := shutdown(ctx); err != nil {
        slog.Error("otel shutdown", "err", err)
    }
}()
slog.SetDefault(slog.New(obs.NewCloudLoggingHandler(ocfg)))

// Root span + run.duration + run.result{status}, same everywhere.
return obs.RunJob(ctx, doWork)

The decisions behind that call apply in any language:

• ComponentType picks the export path. Cloud Run jobs use the in-process Google Cloud exporters: no collector, flush on shutdown. GKE services export OTLP to an in-cluster OpenTelemetry Collector. A service configured without a Collector address fails at startup instead of silently exporting to localhost and dropping everything. Both paths must land on the same metric types, or every template further down breaks; check that first when you add a Collector.
• Sampling follows the workload. Jobs sample every run, because volume is low and every run matters. Services use ParentBased(TraceIDRatioBased(0.1)), so a distributed trace stays complete across hops.
• There's an activation gate. With no GCP project set, or in local mode, the library installs nothing and returns a no-op shutdown, so local runs need no credentials. The W3C tracecontext + baggage propagator is installed either way, so context still flows through a process that isn't exporting.
• A wrapper emits the job contract for you. RunJob opens the root span, times the run, records the outcome and marks the span as errored on failure. Teams can't get the contract wrong because they don't write it.
• The shutdown flush gets its own deadline. For a job, the final flush is the run result. Flush under an already-canceled context (or skip the flush on exit) and you silently drop the one data point every alert depends on.

Porting it to another language

A library in Python, Java or Node.js has the same six jobs. If it does these, the rest of the platform can't tell the difference:

1. Initialize the OpenTelemetry SDK with the Google Cloud exporters for jobs, or OTLP to the Collector for services.
2. Name instruments <service>.<suffix>, with exactly the suffixes, units and label values in the contract below.
3. Wrap the unit of work (a job run, a request) so the contract signals are emitted automatically.
4. Map log fields to what Cloud Logging expects (next section).
5. Flush on shutdown with its own timeout.
6. Publish app events as the JSON shape described in the alert event contract.

The conformance test is simple: run a sample job and check the metric types and labels that appear in Cloud Monitoring. If they match, the dashboards and policies work unchanged.

The logging gotcha that broke Error Reporting

Cloud Logging reads a field called severity, but most JSON loggers write level by default: Go's slog, Python's common JSON formatters, Node's pino, Logback's JSON encoders. Ship that as-is and every line lands at default severity: severity filters, log-based metrics and Error Reporting all quietly misclassify everything.

The library's log handler fixes that in one place:

Two details matter. The trace field needs the full resource name, not a bare trace ID, or the log-to-trace link never appears. And the stack trace is captured at the log call site, written in the language's native stack format (Go's goroutine … [running]: layout for us) so Error Reporting can parse it, with the logging library's own frames dropped. Leave those frames in and Error Reporting groups every error under the logger instead of the code that failed.

That was the cheapest win of the whole project. Once error logs are shaped correctly, Error Reporting groups them fleet-wide with zero configuration. You get "all the problems in one place" on day one, before a single dashboard exists.

The signal contract

A small, fixed vocabulary that every workload emits:

The first three are built into the library. self_heal is a naming convention on top of the generic obs.Count helper, so the library doesn't enforce its label values; code review does.

GKE services emit errors and self_heal too, and get request rate, errors and latency from OpenTelemetry's standard HTTP and gRPC instrumentation (otelhttp / otelgrpc in Go; every language has an equivalent) instead of inventing their own. Pod health comes from GKE system metrics and Managed Service for Prometheus at the platform level, not from the app.

Why the service name is in the metric name

Every instrument is named <service_in_snake_case>.<suffix>, so report-syncer emits report_syncer.run.result, which lands in Cloud Monitoring as workload.googleapis.com/report_syncer.run.result. That's on purpose: the Google Cloud metric type comes from the instrument name, so a prefix guarantees two services can never collide on a metric with a different meaning or shape.

The trade-off is that there's no single run_result metric to query across the fleet. One component's run outcomes look like this:

sum by (status) (
  increase(workload_googleapis_com:report_syncer_run_result[1h])
)

What makes it scale anyway is that the suffixes and labels are identical everywhere. The fleet dashboard, the per-component dashboards and the baseline alert policies are all Terraform templates that loop over the list of components and stamp out the same query for each one. Adding a component means adding a name to a list.

Baseline alert policies

Every component gets two policies by default, so none can be silent:

• Failure: run.result{status="failure"} greater than zero → severity=critical.
• Heartbeat: no run.duration sample within the expected interval (metric absence).

The heartbeat has three edges worth knowing before you rely on it:

1. A metric-absence condition only fires after the metric has existed. A brand-new component that has never written a point can't trip it. Make "first successful run in each environment" part of onboarding.
2. The absence window maxes out at 23.5 hours. That's fine for jobs that run hourly. For a once-a-day job, the schedule plus normal start-time jitter doesn't fit, so write its heartbeat as a PromQL condition over a longer range instead.
3. Every execution is a new time series. The library sets service.instance.id to the Cloud Run execution and task index, so each run writes its own series. Absence is evaluated per series, so aggregate across executions (sum by service) first. Otherwise every finished run's series looks "absent" and the heartbeat fires constantly.

Labels: bounded, and no owners

Service identity travels as resource attributes (service.name, service.namespace, service.version, deployment.environment, service.instance.id and a custom component_type), set once by the library alongside what the GCP resource detector finds. There are deliberately no team or owner labels: routing is by severity, triage context is by service, and ownership metadata goes stale faster than anything else in a label set.

The other hard rule: metric labels stay bounded. status, error_kind, outcome and a few other fixed enums. Never object paths, URIs, run IDs or user IDs: those belong on spans and logs.

Alert policies into Pub/Sub

Most alerts start as Cloud Monitoring policies. Cloud Monitoring already does evaluation, incident lifecycle and its own dedup well, so we let it, and give each policy a Pub/Sub notification channel pointing at the environment's alerts topic.

The setup details that cost us time:

• IAM. Monitoring publishes as its notification service agent, service-<PROJECT_NUMBER>@gcp-sa-monitoring-notification.iam.gserviceaccount.com. It needs roles/pubsub.publisher on the topic. Without it, incidents open in the console and nothing ever arrives.
• User labels carry routing. Every policy sets severity, service and component_type as user labels; the Terraform template sets them, so nobody forgets. They arrive in the incident payload (policy_user_labels), and the dispatcher's Pub/Sub adapter maps them onto the same event shape app code publishes.
• Incidents open and close. The payload carries state: open | closed. Carry that through as a resolved notification for the same incident, not as a new alert, or every recovery looks like a second incident.

The alert event contract

Whether an alert comes from a policy or from code, the dispatcher works with one shape. The contract is the JSON on the wire, not a type in any one language. Our library ships a Go struct for producers, but the dispatcher doesn't import it: the two sides share only the documented schema, so either can evolve on its own as long as the JSON holds. In a polyglot fleet, publish that schema as JSON Schema and generate the types per language:

type Event struct {
    // critical | warning | info
    Severity string `json:"severity"`
    // component name, as in the metric prefix
    Service string `json:"service"`
    // job | service
    ComponentType string `json:"component_type"`
    // one human-readable line
    Summary string `json:"summary"`
    // rare routing override; usually empty
    Channel string `json:"channel,omitempty"`
    // dashboard, trace, logs
    Links map[string]string `json:"links,omitempty"`
    // dedupe key: <service>/<kind>
    Fingerprint string `json:"fingerprint,omitempty"`
    // extra context, never secrets
    Attributes map[string]string `json:"attributes,omitempty"`
}

Two fields do the heavy lifting:

• Severity expresses how much this matters.
• Fingerprint expresses what this is, for example report-syncer/run-blocked, so repeated occurrences collapse into one incident instead of a hundred messages.

There's no environment in the event or the fingerprint, and that's deliberate. Each environment has its own project, topic and dispatcher, so a fingerprint is unambiguous inside the pipeline that receives it.

The producer declares intent, not delivery. It says how severe something is; it doesn't pick a Teams channel, decide whether to email or open a ticket. Routing falls out of severity:

func (e Event) ResolveChannel() string {
    if e.Channel != "" {
        return e.Channel // explicit override, used rarely
    }
    switch e.Severity {
    case SeverityCritical:
        return ChannelBlocker
    case SeverityWarning:
        return ChannelWarn
    default: // unknown severity: never dropped, never pages
        return ChannelDigest
    }
}

The default branch is a small decision we're happy with: a typo in a severity lands in the quietest bucket. The event isn't lost, and nobody gets woken up by a mislabeled one.

ResolveChannel is the producer-side reference for the primary channel. The dispatcher applies the same mapping, and its routing config can add mirrors on top, such as email for criticals.

One hard rule is baked into the contract: the payload never carries sensitive data. No request bodies, no payment fields, no customer identifiers in Summary or Attributes. Alerts travel through Pub/Sub and land in chat tools, which is the wrong place for anything you wouldn't put on a whiteboard.

Pub/Sub in the middle

The topic between producers and the dispatcher is there on purpose:

• Decoupling. Producers don't know or care how many channels exist, or whether the dispatcher is up. They publish and move on.
• One door for two sources. Alert policies and app events arrive the same way, so the dispatcher has one input.
• Fan-out. One event can drive several subscribers: the dispatcher now, an archive or a ticket integration later.

One sharp edge, stated plainly because it bit us in planning: by default, a topic doesn't keep messages; subscriptions do. Only subscriptions that exist at publish time receive a message. Create the topic today and the dispatcher's subscription tomorrow, and everything published in between is gone.

Two ways around it: create the subscription in the same Terraform apply as the topic, or enable topic message retention, which lets a subscription created later seek back to a timestamp and pick up what it missed.

The dispatcher: ports and adapters

The dispatcher turns an incoming event into the right notification: a blockers message, a warnings message, an email or a digest entry, or nothing at all if it's a duplicate. It uses a ports-and-adapters (hexagonal) architecture, so adding a new way to notify never touches the core.

Four boundaries, only three of them ports

Three decisions hide in that table:

• The core doesn't read from Pub/Sub; Pub/Sub calls the core. The core offers one function, Dispatch(ctx, event) error. The Pub/Sub adapter receives each message, calls it, and turns the result into an ack or a nack. The core never sees a subscription, so it's trivial to test and doesn't care how events arrive.
• Normalizing payloads is the adapter's job. Two very different JSON shapes arrive on the topic: Cloud Monitoring incidents and app events. Translating both into one Event is technology-specific parsing, so it lives in the Pub/Sub adapter. The core never sees a Cloud Monitoring payload.
• Static config doesn't need a port. Routing is data that's loaded once, so it's a constructor argument. It would only earn a port if it had to reload at runtime.

The outbound port

package dispatch // the core: no Teams, no SMTP, no URLs

// ChannelID names a configured destination, e.g. "blocker" or
// "oncall_email". It comes from routing config; the core never
// sees URLs or channel types.
type ChannelID string

// State follows the incident: Cloud Monitoring reports open and close.
type State string

const (
    StateOpened   State = "opened"
    StateResolved State = "resolved"
)

// Notification is what the core asks a channel to deliver: what
// happened, not how it looks. Each adapter renders it (Adaptive
// Card, email, ticket).
type Notification struct {
    Key           string // idempotency key: fingerprint + state
    State         State
    Severity      Severity
    Service       string
    ComponentType string
    Summary       string
    Links         map[string]string // dashboard, trace, logs
    Occurrences   int               // repeats folded into this incident
    FirstSeen     time.Time
}

// ErrPermanent marks failures a retry can't fix:
// rejected payload, deleted workflow, revoked credentials.
var ErrPermanent = errors.New("permanent notification failure")

// Notifier is the outbound port. One instance per configured channel.
//
// Contract:
//   - nil means the channel ACCEPTED the notification. Delivery may
//     still be asynchronous (Teams Workflows answers 202 first).
//   - Wrap ErrPermanent when a retry can't help; any other error
//     is transient.
//   - Respect ctx: the core sets a deadline per send.
//   - May be called more than once with the same Key (at-least-once).
//     Adapters that can dedupe on Key should.
type Notifier interface {
    Notify(ctx context.Context, n Notification) error
}

Most of the design lives in the comment, not the signature. What nil means, which errors are worth retrying and whether a duplicate call is safe are exactly the questions Pub/Sub acks and dedupe depend on.

The core, end to end

type DeliveryLog interface {
    // Claim reports whether key should go to ch now: not already
    // sent there within window. Safe under concurrent redeliveries.
    Claim(ctx context.Context, key string, ch ChannelID,
        window time.Duration) (bool, error)
    // Release undoes a claim after a transient failure,
    // so a redelivery retries it.
    Release(ctx context.Context, key string, ch ChannelID) error
}

type Service struct {
    routes   Routes                 // severity → []ChannelID, from config
    channels map[ChannelID]Notifier // wired in main, validated at startup
    log      DeliveryLog
    window   time.Duration
}

// Dispatch is the inbound port.
//   nil           → ack
//   ErrPermanent  → ack and log loudly
//   anything else → nack; the redelivery retries only failed channels
func (s *Service) Dispatch(ctx context.Context, ev Event) error {
    key := ev.Fingerprint + "#" + string(ev.State)
    var transient, permanent []error
    // ev.Channel override, or severity → channels
    for _, ch := range s.routes.For(ev) {
        ok, err := s.log.Claim(ctx, key, ch, s.window)
        if err != nil {
            transient = append(transient, err)
            continue
        }
        if !ok {
            continue // already sent to this channel within the window
        }
        sendCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
        err = s.channels[ch].Notify(sendCtx, s.notification(ev, key))
        cancel()
        switch {
        case err == nil:
        case errors.Is(err, ErrPermanent):
            permanent = append(permanent, fmt.Errorf("%s: %w", ch, err))
        default:
            _ = s.log.Release(ctx, key, ch)
            transient = append(transient, fmt.Errorf("%s: %w", ch, err))
        }
    }
    if len(transient) > 0 {
        return errors.Join(transient...)
    }
    if len(permanent) > 0 {
        return fmt.Errorf("%w: %v",
            ErrPermanent, errors.Join(permanent...))
    }
    return nil
}

Three details carry the weight:

• Claims are per (key, channel), not per fingerprint. A critical goes to both blocker and oncall_email. If Teams succeeds and email fails, the redelivery retries only email. With a single claim per fingerprint you'd either post to Teams twice or lose the email.
• The claim key includes the state. Otherwise a close that arrives inside the window would be swallowed as a duplicate of the open it resolves.
• Errors map onto Pub/Sub. Transient failures nack, so Pub/Sub redelivers. Permanent ones ack, because retrying a rejected payload forever helps nobody. A dead-letter topic with a max-delivery-attempts limit on the subscription is the backstop for anything stuck in between.

The Teams adapter

MS Teams is the first adapter. It posts through Teams Workflows (Power Automate): one workflow per channel, built on the "When a Teams webhook request is received" trigger, which posts the card into that channel.

The trigger expects an Adaptive Card wrapped in a message attachment, and the adapter is also where HTTP status codes become the port's error classes:

// adapters/teams: the only package that knows about Workflows.
type Workflow struct {
    url  string // from Secret Manager, injected in main
    http *http.Client
}

func (w *Workflow) Notify(
    ctx context.Context, n dispatch.Notification,
) error {
    body := message{
        Type: "message",
        Attachments: []attachment{{
            ContentType: "application/vnd.microsoft.card.adaptive",
            Content:     adaptiveCard(n), // opened or resolved layout
        }},
    }
    resp, err := postJSON(ctx, w.http, w.url, body)
    if err != nil {
        return err // network or timeout: transient
    }
    defer resp.Body.Close()
    switch {
    case resp.StatusCode == 400, resp.StatusCode == 401,
        resp.StatusCode == 403, resp.StatusCode == 404:
        return fmt.Errorf("%w: teams workflow returned %d",
            dispatch.ErrPermanent, resp.StatusCode)
    case resp.StatusCode >= 300: // 429, 5xx: transient
        return fmt.Errorf("teams workflow returned %d", resp.StatusCode)
    }
    return nil // 202: accepted, not necessarily posted
}

Three things about Workflows shape the adapter:

• 202 Accepted means queued, not posted. The flow runs asynchronously, so a bad card or a broken flow fails in the flow's run history, not in your HTTP response. That's why the port says nil means accepted. Turn on failure notifications for the flow, and keep the email mirror for criticals so one broken flow can't silence the blockers channel.
• A workflow belongs to a person, not a channel. If its owner leaves and nobody else owns it, the flow stops, and you've rebuilt the silent failure this platform exists to prevent. Add co-owners, and run the flow's connections under a shared or service account.
• The trigger URL is a credential. It carries a signature (sig=) that works like a password, so it lives in Secret Manager, never in config or code.

The other adapters

• Email is a second Notifier, configured as the oncall_email channel: a durable record for criticals that outlives a chat scroll.
• Digest (planned) is a Notifier too. Instead of sending anything, it records the notification for the daily digest job, so info-severity events have a home before the digest exists.
• Tickets (planned) fit the same port, because the contract is incident-shaped: the same Key means the same incident, so the adapter can create a ticket the first time and comment on it after that. It would need its own port only if the core needed something back, for example a ticket URL to put into the Teams card before sending it.

Dedupe: a claim, a window, three kinds of duplicate

Pub/Sub delivers at least once, so the dispatcher will occasionally receive the same event twice. For an alert dispatcher that means a duplicate Teams message at best, and two people jumping on one problem at worst.

It's the same redelivery problem as on the consumer side of any message pipeline (I wrote about it in The Redelivery Trap), and the same family of fix works: claim a key before acting, and skip anything already claimed. Here the claim is per (key, channel), and the state lives in a dedicated schema on a Postgres instance we already ran.

The difference is lifetime. A billing inbox remembers a message forever, because charging twice is never acceptable. An alert dispatcher should remember a claim only for a window:

• Inside the window, a repeat is folded into the existing incident: bump a counter, update "last seen", send nothing.
• After the window, a repeat is news again. A failure that recurs tomorrow should alert tomorrow.

The Postgres side of DeliveryLog.Claim is one upsert:

-- $1 = key (fingerprint#state), $2 = channel,
-- $3 = dedupe window, e.g. '1 hour'
-- PRIMARY KEY (key, channel)
INSERT INTO alert_delivery AS d (key, channel, last_sent_at, repeat_count)
VALUES ($1, $2, now(), 0)
ON CONFLICT (key, channel) DO UPDATE
   SET repeat_count = d.repeat_count + 1,
       last_sent_at = CASE WHEN d.last_sent_at < now() - $3::interval
                           THEN now() ELSE d.last_sent_at END
RETURNING (last_sent_at = now()) AS claimed;

The row lock taken by ON CONFLICT DO UPDATE makes concurrent redeliveries safe: the second one waits, sees a fresh last_sent_at and gets claimed = false. Release sets last_sent_at to '-infinity' for that one channel, so the redelivery claims it again instead of treating it as a duplicate.

That store absorbs three kinds of duplicate: Pub/Sub redeliveries of the same event, a workload reporting the same problem on every run, and the retry after a partial fan-out failure.

One more needs a decision up front: the same failure reported by both sources. If a job publishes its own critical event and the failure policy on run.result fires for the same run, two different payloads describe one problem. Either make them produce the same fingerprint (give the failure policy a kind: run-blocked user label, so both sources produce report-syncer/run-blocked), or decide per condition which source owns it.

Routing and configuration

Delivery is a function of severity and config. The routing table lives in Terraform next to the rest of each environment's infrastructure, not in code and not scattered across per-service channel integrations. The dispatcher reads it once at startup and fails fast if a route names a channel that isn't configured:

# illustrative routing config (one per environment, rendered by Terraform)
channels:
  blocker:      { type: teams_workflow, url_secret: teams-blocker-url }
  warn:         { type: teams_workflow, url_secret: teams-warn-url }
  oncall_email: { type: email, enabled: false } # mirror for criticals
  digest:       { type: digest }                # for the daily digest

dedupe:
  window: 1h   # repeats inside this fold into one incident

routing:
  critical: [blocker, oncall_email]   # act now
  warning:  [warn]                    # checked once a day
  info:     [digest]                  # no real-time ping

Each channels entry becomes one Notifier instance in main; the core only ever sees the keys. Workflow trigger URLs live in Secret Manager and the config only names the secret, so the routing file is safe to review in a pull request.

Severity describes impact, not how much of a run failed. One failed unit can be critical if it's the billing export; half the units failing can be a warning if they're optional or retried on the next run. Only the producer knows which, so the producer decides.

To be precise about the last row: that's info-severity alert events, not info log lines. Logs never pass through the dispatcher.

Because routing lives in config, "stop mailing on criticals in dev" or "add a channel for one noisy component" are config changes, not code changes.

Dashboards: three tiers, all templates

With a uniform signal contract, dashboards become Terraform modules instead of hand-built one-offs. Three tiers let people zoom from "is anything on fire?" to "what exactly broke?":

• Tier 1: Fleet health. The default screen. Rollups and top-N only: open incidents by severity, jobs with no successful run or with failures, service error rate and p95, errors by service, self-heals by outcome.
• Tier 2: By workload class. One for GKE services, one for Cloud Run jobs, grouped by component. For the platform layer we reuse Google's built-in GKE dashboards and the run.googleapis.com/job/* metrics.
• Tier 3: Per component. One deep dashboard per service or job, created from a reusable module when that component adopts the library.

Every Tier 1 or Tier 2 tile links into the relevant Tier 3 dashboard, and every Teams card links to the same places, so an alert, a dashboard and a trace are always one click apart.

Self-healing in code

Once there's a clean "this failed" signal, the tempting move is to wire every failure to an alert. We did, briefly, and promptly rebuilt the noise problem. So workloads now try to recover before anything is reported. The pattern is plain control flow, so it translates to any language:

func (s *Syncer) process(ctx context.Context, unit WorkUnit) Outcome {
    err := s.attempt(ctx, unit)
    if err == nil {
        return Outcome{Status: statusSuccess}
    }

    // 1) Transient? Retry with backoff.
    if isTransient(err) {
        if err2 := s.retryWithBackoff(ctx, unit); err2 == nil {
            obs.Count(ctx, "self_heal", 1,
                attribute.String("outcome", "transient_recovered"))
            return Outcome{Status: statusSuccess}
        }
    }

    // 2) Not transient, or the retry failed: is there a different remedy?
    //    e.g. reload a manifest, reprocess a single file, fall back to an
    //    alternate source. Self-heal doesn't always mean "try again".
    if fix := s.remediation(err); fix != nil && fix(ctx, unit) == nil {
        obs.Count(ctx, "self_heal", 1,
            attribute.String("outcome", "remediated"))
        return Outcome{Status: statusSuccess}
    }

    // 3) Recovery failed: now it's worth a human.
    obs.Errors(ctx, errorKind(err))
    return Outcome{Status: statusFailure, Err: err}
}

Two things to notice. Recovery is failure-specific: a retry for a network blip, a locked row or a cold dependency; a different action for a bad manifest or one corrupt file. The workload knows which remedy fits which failure. And every recovery is counted, so self-healing shows up on a dashboard instead of disappearing. A rising self_heal rate is an early warning in its own right.

Then the run aggregates its outcomes and notifies once, at a severity that reflects the impact of what failed, not the share of units that failed:

fp := func(kind string) string { return svc + "/" + kind }
failed := failedUnits(outcomes)

var ev *alert.Event
switch {
case len(outcomes) == 0:
    // nothing to process: not a failure; the heartbeat policy
    // still catches a job that stops running altogether

case len(failed) == 0:
    // full success: telemetry records it, nobody is interrupted

case s.blocking(failed):
    // downstream data is missing or wrong: interrupt someone
    ev = &alert.Event{
        Severity:    alert.SeverityCritical,
        Fingerprint: fp("run-blocked"),
        Summary: fmt.Sprintf("sync run blocked: %d of %d units failed",
            len(failed), len(outcomes)),
    }

default:
    // failures that can wait: warn, don't page
    ev = &alert.Event{
        Severity:    alert.SeverityWarning,
        Fingerprint: fp("run-degraded"),
        Summary: fmt.Sprintf("sync run degraded: %d of %d units failed",
            len(failed), len(outcomes)),
    }
}

if ev != nil {
    ev.Service, ev.ComponentType = svc, "job"
    ev.Links = runLinks(ctx) // dashboard, trace, logs; never payloads
    if err := s.alerts.Notify(ctx, *ev); err != nil {
        slog.ErrorContext(ctx, "notify failed", "err", err)
    }
}

The impact rule lives in the job, because only the job knows which of its units matter:

// blocking decides by impact, not by ratio: one failed required
// unit (say, the billing export) is critical; a few failed
// optional ones are not.
func (s *Syncer) blocking(failed []Outcome) bool {
    for _, o := range failed {
        if o.Unit.Required {
            return true
        }
    }
    return len(failed) > s.maxOptionalFailures
}

The same rule decides what the run returns: a blocked run returns an error, so run.result records a failure and the baseline policy fires too; a degraded run returns nil and reports only the warning. Otherwise the policy would page for runs the job itself rated as able to wait.

The empty-run case is spelled out because "nothing to do" and "nothing worked" must never look the same. Ignoring an empty run is safe here only because the heartbeat policy covers a job that stops running.

The Notify error is logged, not fatal. The failure policy on run.result is the second, independent path to the blockers channel, which is exactly why the both-sources dedupe case above matters.

What's next

• A daily digest. A scheduled Cloud Run job that assembles the last 24 hours (active incidents, failures and missed runs, service errors and latency, top Error Reporting groups, self-heal counts, GKE restarts) and emails it delta-first: what's new since yesterday comes before what's still true. "Nothing to report" is a good digest.
• Tickets. The ticket adapter described above, switched on per environment from a configurable severity.
• AI-assisted triage, longer term. We're exploring plugging Aura Tracker GCP, an open-source GCP investigation tool I maintain, into the pipeline: analyzing incidents against live cloud state and writing a likely cause and recommended fix into the ticket. Applying fixes, if ever, only behind a preview-and-approve step.

Design decisions and trade-offs

If you're building something similar

Start in this order; each step is useful on its own:

1. Fix log shape first. severity, message, full trace resource name, stack_trace on errors. Error Reporting alone will show you more on day one than most dashboards.
2. Define the signal contract and emit it from a wrapper, not by hand.
3. Add the two baseline policies per component: failure and heartbeat, with the 23.5-hour limit in mind.
4. Point policies at a Pub/Sub topic, grant the Monitoring service agent publisher on it, and put routing in user labels.
5. Build the dispatcher last, with per-channel dedupe and a clear transient vs. permanent error contract from the first commit.
6. Then teach workloads to heal themselves, and count every time they do.

The dispatcher, the routing and the self-healing all get easier once "telemetry" and "notification" stop being the same word.