跳到正文
DEV Community· SilasFletcher5857·· 2 小時前AI 評分58

錯誤追蹤與可用性監測無法取代 Cron 完成心跳

Error Tracking and Uptime Monitoring Expose Different Cron Heartbeat Evidence

AI 導讀

錯誤追蹤與可用性監測無法證明排程工作已完成,作者主張以外部評估的完成心跳偵測未啟動或逾期的工作。文章建議為每次排程建立穩定的 occurrence ID,待業務輸出及延遲、成本數據持久化後才記錄完成,並在寬限期過後為未完成的預期工作發出通知。

正文

Error tracking and uptime monitoring cannot prove that a media cron job sent its completion heartbeat. An AI agent that never starts produces no exception, no latency sample, and no cost record, so scheduled work needs an external deadline check.

TL;DR: use error tracking for failures observed by running code, uptime monitoring for reachable services, and an externally evaluated heartbeat for scheduled work that might never run. Give every scheduled agent loop a stable occurrence ID, record completion only after its durable output and measurements are committed, and page on a missing deadline rather than on a quiet error stream.

After pages for both missed jobs and duplicate deliveries, I no longer treat a quiet error stream as evidence of health. The runbook lesson is the same for both: retries and monitoring must agree on the identity of one scheduled occurrence. An error event proves that some code saw a failure. A completion heartbeat proves that expected work reached a defined terminal state. Neither claim contains the other.

Can Error Tracking or Uptime Monitoring Replace a Cron Heartbeat?

Picture a publisher that runs an agent after each editorial batch. The agent classifies stories, records model latency and cost, then writes an enrichment result. Its web service can answer an uptime probe throughout the night. Its error tracker can remain empty too. If the scheduler never dispatches the 02:00 occurrence, there is no process in which to throw an exception.

That is a silent failure, not a successful zero-work interval. An empty latency chart could mean that no calls were due, telemetry stopped, the queue stalled, or the run never began. Likewise, a zero cost total says nothing about scheduling health. Metrics from completed calls describe calls that happened; they cannot prove an expected call existed.

Silence proves nothing.

The invariant is blunt: absence must be judged by a clock outside the job. If the same worker both performs the task and decides whether its missing completion is alarming, one failure can remove the work and its witness. I used to begin these pages by looking for an exception. A missed job taught me to start one step earlier: find the expected occurrence, then ask which transition is absent. No occurrence record points toward scheduling. A started record without completed points toward the worker or its dependencies. A completed record with no accepted output is a transaction problem. Those are different response paths, and forcing them all through an error dashboard wastes the first minutes of an incident.

This is also why a start ping is insufficient. It proves dispatch. For a loop that fans out across a media batch, completion should mean that the accepted output and its latency and cost record are durable. If partial results are valid, define that terminal state before the incident. A green ping cannot invent the policy afterward.

Give Each Signal One Narrow Claim

The three signal types overlap on a dashboard, but their evidence differs.

Signal What it establishes What it can miss Useful first response
Error tracking Executed code observed a crash, thrown exception, or explicitly captured error A process that never started; a valid but wrong early return Inspect stack, context, and release
Uptime monitoring A target responded under the probe's conditions A stopped scheduler or worker while the API remains reachable Check the service and network path
Completion heartbeat A named occurrence checked in before its deadline Semantically bad output from a completed run Inspect scheduler, queue, worker, and occurrence ID

Signal quality depends on refusing to overload those claims. Sending an error event for a missed deadline fabricates an application failure that no application observed. Treating a successful health endpoint as proof of batch completion hides independent failure domains. Calling every late run immediately, meanwhile, creates enough noise that operators stop trusting the page.

That page gets muted.

The grace period should cover normal execution time, queue delay, and the retry policy. Do not copy the five-minute schedule into a five-minute alert threshold by reflex. Use the observed distribution of successful runs, add the operational margin your response policy requires, and revisit it when the workload or model route changes. There is no universal duration in this advice.

Make the Preventative Path Idempotent

The occurrence identity should be deterministic: job name plus scheduled time is often enough. A retry that generates a fresh random ID looks like new work, which can produce both duplicate publication and duplicate telemetry. The completion write should accept the same result twice, yet reject a second result with a different digest for the same occurrence. This isn't bookkeeping. It is the difference between one delayed edition and two published editions.

The capture path below sends an error that running code has actually observed. The JSON body comes from INFRAI_ERROR_CAPTURE_JSON because the live discovery schema, rather than an article, should define its fields. The client uses the verified route, an environment credential, an explicit method, response-body errors, and bounded exponential backoff that honors Retry-After on HTTP 429. It does not send a capture for a missing run; the external heartbeat evaluator owns that case.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func delay(retryAfter string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    body := []byte(os.Getenv("INFRAI_ERROR_CAPTURE_JSON"))
    if key == "" || len(body) == 0 {
        panic("INFRAI_API_KEY and INFRAI_ERROR_CAPTURE_JSON are required")
    }

    endpoint := "https://api." + "infrai" + ".cc/v1/errors/capture"
    client := &http.Client{Timeout: 10 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(delay(resp.Header.Get("Retry-After"), attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            panic(fmt.Sprintf("capture returned %s: %s", resp.Status, strings.TrimSpace(string(responseBody))))
        }
        fmt.Println("error captured")
        return
    }
    panic("rate limit persisted after four attempts")
}

Keep the heartbeat path separate. Its order of operations is: establish the expected occurrence independently, commit the business result, record its measurements, then mark completion. In a real transaction, the durable result and completion state should not disagree after a crash.

Run the overdue evaluator elsewhere. It should notify only when an expected record exists, its deadline plus grace period has passed, and no accepted terminal state is present. That condition avoids unconditional “nothing happened” pages during an intentional pause. It also gives the responder a concrete ID to follow through scheduler, queue, agent call, and publication records.

Compare Products by Missing Evidence

Error trackers and heartbeat monitors are usually complements. Sentry is oriented toward application errors and rich debugging context. Healthchecks.io watches cron-style jobs through success, start, failure, and log signals. Cronitor also focuses on cron jobs and scheduled-task monitoring. Better Stack offers heartbeat monitoring alongside uptime checks. Datadog provides synthetic tests within a broader monitoring platform.

The fair choice follows the evidence gap. Sentry plus Healthchecks.io is a sensible small-system pairing when exception investigation and missed-job detection are separate needs. Cronitor fits teams that want scheduled jobs to be the center of the operational view. Better Stack can reduce tool switching when heartbeats and endpoint uptime belong together. Datadog is more natural when synthetic checks need to join an existing, wider telemetry estate. Verify current notification channels, retention, and plan boundaries in each product's documentation before standardizing; those details change.

Infrai's relevant advantage is one API key, one wallet, and one bill across 295 routes in 20 modules. Its single REST API avoids stitching together 30 SDKs, juggling 30 keys, or reconciling 30 invoices as the media system adds backend capabilities. Error capture can record failures from code that executes, but it does not supply synthetic checks, heartbeats, or missed-task alerts, and its observability surface has no notification route. Pair it with a Healthchecks-style service or an independently operated poller and notifier. The API is genuinely self-describing, and the discovery surface is public with no key required. Its platform convention marks 171 of 294 capabilities as idempotent and specifies a 24-hour default deduplication window, useful limits to inspect before retries go live. Those traits make contract inspection and additional backend calls straightforward, but they cannot turn captured errors into proof that a schedule ran.

There are sharper boundaries too. Its logs can carry trace_id and span_id, but it has no distributed trace query or span tree. It does not provide source-map decoding, crash symbolication, Electron minidump parsing, or session replay. A product built for rich crash reconstruction or trace navigation is the better choice when those are the deciding requirements.

No vendor removes the semantic work. The team must define when the media output is accepted, which occurrence owns a retry, and how late is late enough to page.

When Is Error Tracking Alone Enough?

Error tracking alone is a reasonable starting point for a request-driven application whose requirement is exception visibility inside code that runs. A heartbeat adds little when there is no expected schedule or deadline. An uptime check may also be enough when its successful response exercises the complete user-visible operation rather than a shallow process check.

Custom polling is justified when completion semantics are unusually specific. Create the expected-run record before dispatch, update it idempotently at an accepted terminal state, and query overdue records from another process. The trade-off is ownership: your team now operates deadline evaluation, notification delivery, deduplication, escalation, and clock policy.

My default runbook rule is conservative. Page only when a named occurrence was expected, its evidence-based grace period expired, and completion is absent. Route exceptions to error tracking, use uptime checks for externally reachable behavior, and keep latency and cost attached to the same occurrence ID. This preserves the meaning of every signal and keeps ordinary variance from drowning the one silence that matters.

Sources

來源:DEV Community · dev.to