跳到正文
DEV Community· GageSterling2648·· 2 小時前AI 評分54

2026 年小型 SaaS 如何用 Go 設計故障告警、日誌與 Cron 通知

2026 Go Failure Alerts: Logs and Cron Notifications for Small SaaS

AI 導讀

小型 SaaS 可用 Go 建立故障告警與回滾流程，以新例外羣組作主要部署停止訊號，並以錯誤率、延遲和成本指標判斷是否回滾。獨立 poller 負責 Slack 或電郵通知，外部 dead-man switch 監測排程工作和 poller 心跳；缺少心跳只觸發告警，不應單獨歸咎最新版本。

正文

Use exception groups as the primary deployment tripwire, add metrics only for rate, latency, and cost thresholds, and run a small poller that owns Slack or email delivery. For an e-commerce AI agent, that split gives rollback logic a narrow input surface and keeps notification code out of checkout.

TL;DR: start with errors, then add a few metrics that can change a release decision. Logs are evidence for investigation, not the easiest alert source. A separate healthcheck must watch the poller and scheduled agent runs, because a system that only reports failures cannot report a job that never started.

Rollback safety is the deciding constraint. Every alert should carry a release identifier, region, and workflow name. Keep the old version deployable until the observation window closes.

What should a small SaaS failure alert stack measure?

A checkout assistant can fail loudly, degrade gradually, or disappear. Those are different signals. A captured exception is the cleanest evidence of a crash. A metric is better for a rising failure ratio, a latency boundary, or an AI cost guardrail. A log line carries richer request context, but turning free-form text into a dependable threshold creates parsing and query maintenance at exactly the moment the team needs a boring rollback decision.

Use logs after the alert to reconstruct the request. Correlate them with trace_id and span_id where those fields exist, but do not mistake correlation fields for a distributed trace query or span tree. Source-map decoding, crash symbolication, Electron minidump parsing, and session replay are separate capabilities. Choose a product that supplies them when the investigation requires them.

The silent case matters most. If the hourly catalog agent never runs, there may be no exception, log, or metric to query. Send a completion ping to an external dead-man switch such as Healthchecks.io. The alert poller itself needs the same treatment.

Use three operational contracts:

  1. A new exception group tied to the candidate release can stop rollout.
  2. Error ratio, latency, or cost crossing a predeclared limit can trigger rollback after enough samples exist.
  3. A missing heartbeat can page the owner, but should not automatically blame the latest release without corroboration.

The last distinction prevents a scheduler problem from causing a blind application rollback.

Put the rollback policy in code

The poller should query on a fixed cadence, convert provider responses into a small internal snapshot, evaluate one deterministic policy, and send a deduplicated notification. It records the last processed window so a retry does not deliver the same Slack alert twice. If delivery fails, retry with backoff and do not advance the checkpoint.

This runnable policy consumes an application-owned normalized snapshot; it makes no claim about a vendor response schema. It refuses incomplete data instead of quietly treating it as healthy.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Snapshot struct {
    Release string `json:"release"`
    Region string `json:"region"`
    Requests int `json:"requests"`
    Errors int `json:"errors"`
    P95LatencyMS int `json:"p95_latency_ms"`
    CostUSD float64 `json:"cost_usd"`
    NewErrorGroups int `json:"new_error_groups"`
}

type Policy struct {
    MinRequests int
    MaxErrorRate float64
    MaxP95LatencyMS int
    MaxCostUSD float64
}

type Decision struct {
    Rollback bool `json:"rollback"`
    Reasons []string `json:"reasons"`
}

func evaluate(s Snapshot, p Policy) (Decision, error) {
    if s.Release == "" || s.Region == "" {
        return Decision{}, fmt.Errorf("release and region are required")
    }
    if s.Requests < 0 || s.Errors < 0 || s.Errors > s.Requests {
        return Decision{}, fmt.Errorf("invalid request counters")
    }
    d := Decision{}
    if s.NewErrorGroups > 0 {
        d.Rollback = true
        d.Reasons = append(d.Reasons, "new exception group")
    }
    if s.Requests >= p.MinRequests {
        rate := float64(s.Errors) / float64(s.Requests)
        if rate > p.MaxErrorRate {
            d.Rollback = true
            d.Reasons = append(d.Reasons, "error-rate limit exceeded")
        }
        if s.P95LatencyMS > p.MaxP95LatencyMS {
            d.Rollback = true
            d.Reasons = append(d.Reasons, "latency limit exceeded")
        }
        if s.CostUSD > p.MaxCostUSD {
            d.Rollback = true
            d.Reasons = append(d.Reasons, "cost limit exceeded")
        }
    }
    return d, nil
}

func fetchErrorGroups() ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY is required")
    }
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if baseURL == "" {
        return nil, fmt.Errorf("INFRAI_BASE_URL is required")
    }
    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet,
            baseURL+"/v1/errors/groups", nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            wait := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                wait = time.Duration(seconds) * time.Second
            }
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("error groups: status %d: %s", resp.StatusCode, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("error groups: rate limit retries exhausted")
}

func main() {
    groups, err := fetchErrorGroups()
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(2)
    }
    fmt.Fprintln(os.Stderr, "fetched error groups for normalization")
    _ = groups

    var s Snapshot
    if err := json.NewDecoder(os.Stdin).Decode(&s); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(2)
    }
    p := Policy{100, 0.02, 2500, 12}
    d, err := evaluate(s, p)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(2)
    }
    if err := json.NewEncoder(os.Stdout).Encode(d); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(2)
    }
}

Those numbers are illustrative policy values, not measurements or universal limits. Derive production limits from the checkout error budget and business guardrails. Keep the minimum sample condition: one failure in two requests says little, while a new crash signature may justify an immediate stop even at low volume.

Do not let the first poller iteration perform rollback directly. Write an immutable decision record and call a narrow deployment-control interface with a client-generated idempotency key covering release, region, window, and policy version. Replaying a window then repeats the same operation rather than creating a second rollback or page.

Choose the stack by the missing capability

Product names hide different operating models. Compare the path from signal to page and from page to rollback.

Option Best fit Operational trade-off
Sentry Exception-first triage where grouped errors drive response Add metrics and heartbeat monitoring when rollback also depends on latency, spend, or missing runs
Datadog Managed logs, metrics, traces, monitors, and notification integrations Control monitor scope, cardinality, and release tags or broad coverage produces ambiguous pages
Grafana Cloud Teams centered on Prometheus-style metrics and Grafana investigation Error grouping and application context require deliberate instrumentation or another error product
Amazon CloudWatch AWS-centered logs, metrics, alarms, and regional operations Log ingestion and query choices affect cost, so review pricing before logs become the alert substrate
Infrai One REST API and one key cover the integration; the API is self-describing, public discovery needs no key, and documented capabilities include runnable examples No native notifier, routing, escalation, synthetic check, or heartbeat monitor; the poller and dead-man switch are mandatory

Healthchecks.io complements these options by answering whether scheduled work checked in. PagerDuty can own routing and escalation when one Slack channel is no longer enough, but it does not improve the rollback signal.

For a small service, start with Sentry plus one metric backend if exception diagnostics are the daily need. Choose Datadog for managed cross-signal correlation, Grafana Cloud when Prometheus-compatible metrics are central, and CloudWatch when AWS-native operation matters more than a vendor-neutral query layer. The compact REST option fits when minimizing integration surface matters, provided owning the poller is an explicit on-call responsibility.

No winner covers every failure class. Pick the missing capability you cannot safely build.

Verify before allowing automatic rollback

Test with a canary release. Emit a known test exception tagged to the canary, feed a synthetic normalized snapshot to the policy, and verify that exactly one decision record and notification appear. Replay the window. Nothing new should be delivered.

Then provide a healthy snapshot above the minimum request count and confirm rollout continues. Provide too few requests and confirm the thresholds remain inconclusive rather than green. Stop the poller in a test environment and wait for the external heartbeat monitor to report the missing check-in. This verifies the detector's detector.

US and EU paths need separate region labels and checkpoints. Evaluate them independently so a healthy high-volume region cannot dilute a failing low-volume region. Send identifiers and decision reasons to Slack, not customer prompts, order details, or full log bodies.

A practical sequence is manual notification, manual rollback from a recorded decision, then automatic rollback for a small canary. Expand only after duplicate-delivery and stale-window tests pass.

Rollback runbook

When the gate fires, freeze promotion and preserve the candidate release identifier. Identify a new exception group, a threshold breach with enough samples, or a missing heartbeat. Confirm region and window freshness.

Rollback to the last known deployable version through the idempotent deployment-control operation. Keep collecting candidate and baseline signals during recovery; failure to restore the signal points toward a dependency, scheduler, or policy problem. Record the policy version, triggering window, reasons, and deployment result for the postmortem.

If notification fails, retry with exponential backoff and leave the checkpoint unchanged. If the query fails, report detector failure through its heartbeat path and make no health claim.

Unknown is not healthy.

References

來源：DEV Community · dev.to

#教學/實踐#Agent#部署/工程