綠色勾號並不可靠：Cron 工作、工作程序與無聲 Web API 的後端錯誤追蹤

Short answer: use separate signals for an exception, a missed schedule, an unfinished run, and an invalid result. Exception capture is necessary for cron jobs, workers, and web APIs, but it cannot see work that never starts or output that is wrong while the process exits normally.

That distinction is the starting point for an error-tracking design I can evaluate from a notebook through production. The simple approach is one try/except around each entry point. The more useful approach gives every run a lifecycle, checks the application contract, and sends searchable context without copying prompts or customer data into an incident system.

What should error tracking observe across cron jobs, workers, and web APIs?

Think in four questions: did execution start, did it finish, did code raise, and was the result acceptable? A scheduler heartbeat answers the first. A completion signal answers the second. Exception capture answers the third. A schema or domain assertion answers the fourth. They can share a run ID and an incident view, but they should not be collapsed into one “error” counter.

This model catches silent failures that ordinary logs miss. A cron process may never be invoked after a schedule change. A queue worker may start and stop renewing its lease. A web API may return 200 with an empty retrieval set or malformed JSON. None of those states requires a Python exception.

The four golden signals from the Google SRE guidance still provide the service-level frame: latency, traffic, errors, and saturation. Scheduled work adds an expectation about time, while an AI endpoint adds an expectation about answer quality. An eval harness should check a fixed dataset and threshold outside the request path; production telemetry can tell me that the request was fast and valid, but not that its answer met the retrieval contract.

No single signal wins.

The experiment: why exception-only monitoring loses the important cases

I compare designs with failure experiments before choosing an implementation. In the first experiment, the entry point catches an exception and re-raises it. That is good for a failed database call or a KeyError, and re-raising preserves the runtime's retry behavior. It says nothing about a scheduler that did not launch the process.

The second experiment skips a run deliberately. An independent deadline monitor should alert after the allowed grace period, even though the application emitted no log line. The third blocks a worker after its start signal; the monitor should report “started without completion” once, with the run ID and deadline. The fourth feeds a response with the wrong shape. A contract assertion should reject it before a successful HTTP status can hide the problem.

I keep the evidence small and deliberate. Operation name, release, trace or run ID, exception type, elapsed milliseconds, and safe shape metadata are useful search keys. Full prompts, retrieved documents, API keys, and model responses are poor default payloads: they increase exposure and storage cost while making grouping less precise. For a RAG pipeline, a prompt version, dataset version, token count, and eval case ID usually explain more than the text itself.

Here is the decision table I use when reviewing an alert policy:

I'm not sure every empty result is a failure; your mileage may vary when “no rows” is a valid answer. That uncertainty belongs in the contract and its tests, not in a vague dashboard label.

A Python boundary that keeps lifecycle and exception signals distinct

The application can depend on two tiny adapters: one sends lifecycle signals, and one accepts structured exception events. Those adapters can target a self-hosted receiver, a managed telemetry service, or a logging pipeline. The worker remains testable because neither destination is embedded in its business logic.

from __future__ import annotations

import json
import time
import traceback
import urllib.request
from collections.abc import Callable
from typing import Any


EventSink = Callable[[dict[str, Any]], None]
SignalSink = Callable[[str, str], None]


def send_signal(base_url: str, state: str, run_id: str) -> None:
    payload = json.dumps({"run_id": run_id}).encode("utf-8")
    request = urllib.request.Request(
        f"{base_url}/{state}",
        data=payload,
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=5) as response:
        if response.status >= 400:
            raise RuntimeError(f"lifecycle signal rejected: {response.status}")


def run_job(
    run_id: str,
    fetch_records: Callable[[], list[dict[str, Any]]],
    index_records: Callable[[list[dict[str, Any]]], int],
    capture: EventSink,
    signal: SignalSink,
) -> None:
    started = time.monotonic()
    signal("start", run_id)
    try:
        records = fetch_records()
        indexed = index_records(records)
        if indexed != len(records):
            raise ValueError("index count did not match input count")
        signal("complete", run_id)
    except Exception as error:
        capture(
            {
                "kind": type(error).__name__,
                "operation": "index-records",
                "run_id": run_id,
                "elapsed_ms": round((time.monotonic() - started) * 1000),
                "stack": traceback.format_exc(),
            }
        )
        raise

The pseudonymous URL is an application contract, not a vendor route. I send complete only after the count assertion passes; putting it in finally would mark a failed run healthy. The exception escapes after capture so the process supervisor or queue can apply its normal retry policy. In a web API, the same boundary sits around request handling, with the trace ID propagated to downstream calls.

Tests make the behavior concrete. Inject a capture list and a signal list. A fixture that raises from fetch_records must produce an exception event and no completion signal. A fixture that returns three records while indexing two must raise ValueError, produce the same event fields, and still omit completion. A separate scheduler test omits start and verifies that the deadline monitor creates one actionable alert rather than waiting for an exception.

Fast feedback first.

Search, resolve, and operate the signal instead of collecting it

An event is useful only when an operator can move from symptom to owner. Search should filter by operation, release, run ID, exception type, and a bounded set of tags. Grouping should merge repeated instances of the same cause without hiding a new release or a different assertion. Resolution should record who accepted the fix and link to a regression test or an eval case.

For cron jobs, model the expected window and grace period explicitly. “No event for six hours” is weaker than “the 02:00 run has no start by 02:15.” For workers, retain queue name, attempt number, and lease deadline. For APIs, preserve trace propagation and distinguish expected 4xx client responses from server-side failures. Health checks should test dependencies and freshness, not just process liveness.

Custom appenders and transport hooks follow the same adapter pattern. The Logback appender documentation shows how an event can be handed to a custom destination; the language changes, but the design concern does not. A custom sink owns buffering, timeouts, redaction, backpressure, and shutdown behavior. That work is worthwhile only when the existing telemetry path cannot meet policy or correlation requirements.

Where this layered design is not the right fit

The catch is operational overhead. A small, low-impact internal job may be better served by platform logs and a native scheduler alert, especially when one team already owns that dashboard. A regulated deployment may need a self-hosted pipeline so stack traces and identifiers stay inside its boundary. A high-volume service whose main problem is cross-service latency should prioritize standard traces and metrics, with exception events attached rather than duplicated into a second alert stream.

Sampling also needs discipline. I can sample repetitive successful traces after measuring their value, but I would keep rare exceptions and missed schedules unsampled until their frequency and impact are understood. Token and retention budgets matter for AI systems, yet detection quality and data handling come first; a smaller, redacted event is preferable to a cheap archive of sensitive prompts.

Before copying this design, measure missed-run detection time, exception delivery delay, duplicate-alert rate, searchable-context completeness, and the fraction of alerts that lead to a concrete action. Then run the failures on purpose. A quiet dashboard is not evidence.

References

• Google SRE Book, “Monitoring Distributed Systems”: https://sre.google/sre-book/monitoring-distributed-systems/
• Logback Manual, “Appenders”: https://logback.qos.ch/manual/appenders.html