異常尚不等於訊號：產品團隊的實用檢視流程

A dashboard changes. A metric moves in the wrong direction. Someone says, "Something looks different."

That observation might be the beginning of an important discovery. It might also be seasonality, instrumentation drift, a temporary release effect, a small sample, or a perfectly ordinary change in customer mix.

The difficult engineering question is not how to detect more anomalies. It is how to decide which unexpected observations deserve investigation, without turning every dashboard movement into an incident or a new product roadmap item.

This article proposes a small, testable review loop. It is an editorial design proposal, not a validated Cralgo method or a claim about customer results.

Three different queues, three different decisions

Teams often put unlike things into the same Slack channel.

An incident is a condition requiring operational response under an agreed reliability, safety, security or customer-impact policy. If a service-level objective is being breached or customer harm is occurring, follow the incident process. Do not wait for an exploratory research meeting.

An anomaly is a departure from an expected pattern. It is a description of data, not yet an explanation of cause or importance.

A candidate signal is an anomaly or qualitative observation with enough context to justify a specific question. It is still not a finding, prediction or validated trend.

This distinction matters. Google’s SRE guidance ties paging decisions to meaningful service impact and error-budget consumption, rather than simply treating any unusual measurement as urgent. That is a useful boundary for an exploratory signal process, not a reason to weaken incident alerting.

Practical rule: Safety, privacy, security and agreed incident thresholds override the signal queue. The signal queue is for learning when immediate operational response is not already required.

The hypothetical checkout anomaly

Imagine a consumer app. Its checkout completion rate appears to fall over a weekend, while overall sessions increase.

A hasty reading: "The new checkout has made conversion worse."

A second hasty reading: "Traffic quality has deteriorated."

Neither follows from the dashboard alone.

The team should first ask:

• Did the denominator change? Are we comparing sessions, checkout starts or eligible users?
• Did instrumentation change? Was an event renamed, delayed, deduplicated or lost?
• Did channel, geography, device or new-versus-returning customer mix change?
• Was there a price, stock, delivery, payment or promotion change?
• Is the apparent shift large relative to ordinary variation and sample size?
• What would we expect to see if each explanation were true?

The observation is worth preserving even if it turns out to be a tracking issue. But its status should remain unresolved, not "customer behaviour changed."

A compact candidate-signal record

A signal record should be small enough that a team will actually complete it. Here is one possible structure:

id: SIG-EXAMPLE-001
observed_at: "2026-10-24"
observation: "Checkout completion appears lower during a traffic increase"
source: "Aggregate funnel dashboard; event-definition version pending"
baseline: "Compare matched periods and eligible cohorts before interpreting"
competing_explanations:
  - "Event instrumentation or denominator changed"
  - "Traffic or customer mix changed"
  - "Actual checkout friction increased"
next_discriminating_check: "Reconcile events and segment by source/device"
decision_owner: "Product analytics lead"
status: "unresolved"

The date and ID are illustrative. This is not a real Cralgo or client dataset.

Notice what is missing: a certainty score, a projected revenue gain, and an automatic recommendation to rebuild checkout. The record carries a question and a next check, not a conclusion.

The minimum useful fields are observation, source, baseline, alternative explanations, next discriminating check, owner, and status. A team can extend them with expected impact, privacy classification, review date and links to approved aggregate evidence.

Instrumentation is necessary; interpretation is separate

OpenTelemetry provides conventions for recording traces, metrics and logs consistently across services. Consistent instrumentation makes technical investigation easier, but telemetry cannot by itself determine what an organisational or customer observation means.

Consider two event streams:

1. System telemetry: request errors, latency, service dependencies, payment failures and release identifiers.
2. Product events: checkout starts, address validation attempts, payment-method selection, successful orders and user-visible error states.

Neither stream should be treated as the whole customer experience. A rise in checkout starts may be caused by a new marketing campaign. A fall in payment success may be specific to one provider. A decrease in a button-click event may be a frontend instrumentation change.

The goal is not to collect everything. It is to preserve enough reliable context to distinguish plausible explanations. Minimise personal data, respect consent and retention rules, and prefer aggregate or privacy-preserving analysis when individual-level data is unnecessary.

Give every observation a route to closure

A signal register becomes a graveyard if entries only accumulate. Every candidate should end in one of four states:

Escalated: The evidence meets an incident, safety, security or other operational threshold. Hand it to the appropriate established process.

Investigating: The team has a named owner, a discriminating check and a review date.

Learned: A supported explanation was established, with source links and remaining uncertainty recorded.

Closed without conclusion: The signal was too weak, too costly to investigate or no longer relevant. Preserve the reason; do not rewrite it as disproven.

A fifth outcome may be converted to an experiment, but only after the team can state a hypothesis, a measurable outcome and a plausible way to distinguish an intervention effect from background variation.

This prevents an attractive story from silently becoming a product requirement.

One weekly review, not another dashboard meeting

A small product team could trial a 25-minute weekly review:

Five minutes — triage. Remove items that belong in incident response. Merge duplicates. Reject observations without a source.

Ten minutes — challenge. Select at most two candidates. Ask for the strongest alternative explanation and the cheapest check that could disconfirm the preferred story.

Five minutes — decide. Assign an owner and a deadline, or close the item explicitly.

Five minutes — learn. Revisit last week’s decisions. Which observations survived scrutiny? Which were measurement artefacts? Which actions were changed by evidence?

The timebox and item limits are suggested trial parameters, not researched optimal values.

The review should not reward the number of signals logged. That would encourage more logging, not better judgement.

How would we know this loop is useful?

Run it in shadow mode for four weeks before making it part of a formal operating process. Do not change incident paging, release controls or privacy policy.

Compare the review’s outputs with the team’s existing approach:

• Traceability: Can another person reconstruct why a candidate was raised and what happened next?
• False escalation: How often did a candidate trigger unnecessary urgent attention?
• Decision relevance: Did any investigation materially change a decision, or only produce commentary?
• Time cost: How much analyst, engineer and product time did the loop consume?
• Missed importance: Were consequential observations overlooked because they were not easy to quantify?

These are proposed evaluation questions, not reported results. It is possible that the loop adds overhead without improving decisions. If so, simplify or stop it.

Google’s guidance on blameless postmortems offers a related lesson for incidents: records become valuable when they preserve evidence, ownership and learning, not when they merely document activity. An exploratory signal review should hold itself to a similarly practical standard while remaining separate from incident management.

The counterargument: perhaps teams already have too many review processes

That objection is serious.

A new register can make ordinary work slower. Teams may start seeking permission to investigate. Senior leaders may treat unresolved questions as weak performance. Analysts may be asked to explain every fluctuation. Documentation can grow faster than understanding.

The answer is not a more elaborate framework.

Try the smallest possible review against a concrete decision. Keep operational alerts outside it. Preserve alternative explanations. Close weak items. Stop if it does not change what the team can understand or decide.

An anomaly becomes worth pursuing not when it looks interesting on a graph, but when there is a specific question, a plausible way to test it, and a reason the answer might matter.

That is the boundary between observing more and learning more.

Sources and further reading

• Google SRE Workbook, Monitoring — monitoring purposes, telemetry trade-offs and alert severity.
• Google SRE Workbook, Alerting on SLOs — actionable operational alerts and significant service impact.
• OpenTelemetry Semantic Conventions — consistent technical event, trace, metric and log naming.
• Google SRE Workbook, Postmortem Culture — traceable evidence, ownership and organisational learning.
• Cralgo Signals, The Signals We Overlook — the broader editorial question of distinguishing an observation from a conclusion.

Disclosure: This article presents a proposed review practice informed by published technical guidance and Cralgo's conceptual editorial work. It does not report an empirical study, a customer deployment or validated performance improvements. AI assisted with research synthesis and drafting. This is a conceptual proposal, not an empirical result.