FastAPI 僅靠即時發布無法可靠送達通知：以遊標回補保留事件

TL;DR: Realtime publish alone cannot deliver notifications reliably because a live send does not prove that an event became durable, was observed, or can be recovered after a disconnect. For a gaming operations dashboard, persist every device-status event under a stable ID, then treat the live channel as a low-latency hint. On reconnect, the client presents its last applied cursor and reads the missing range before resuming the live tail.

That conclusion comes from a blunt evaluation constraint. Disconnect a dashboard, emit several console status changes, reconnect it, and require the final screen to match the stored event history without duplicates. A publish-only design looks excellent while the tab stays connected. It cannot pass the gap test.

The chosen design has two paths with different jobs: a durable read path establishes truth, while a transient push path reduces display latency. The inbox is the record; the stream is the wake-up call.

Why can't realtime publish alone deliver notifications?

Publish answers a narrow question: could the system attempt to send this payload to subscribers visible at that moment? Reliable notification delivery asks more. Was the event committed before it was announced? Which dashboard may read it? Has this client applied it? Can another client recover it later? What happens if the network drops after send but before render?

A connection-oriented transport cannot answer those application questions by itself. WebRTC, for example, specifies peer connections and data transport. It does not define an application inbox, retention policy, per-user read cursor, or replay contract. Those belong in the application architecture. The same separation applies when the live hop uses a different transport.

Acknowledgements also leave ambiguity. If a browser acknowledges event 418 and that acknowledgement is lost, the server cannot distinguish it from a browser that never applied event 418. Retrying may duplicate the update; refusing to retry may leave a hole. Stable event IDs make that ambiguity manageable because applying the same event twice can be harmless.

This matters in a gaming fleet. A dashboard might show a console moving through online, degraded, and offline while an operator's laptop changes networks. Showing only the newest state can be fine for a fleet map, but it is wrong for an alert timeline that must preserve the transition into degradation. Reliability starts with a product decision: is the reader consuming current state, ordered history, or both?

The two-plane inbox model

The write sequence is critical. First, validate and authorize the status change. Next, commit an immutable event to durable storage and update any current-state projection within the application's chosen consistency boundary. Only after that commit should a publisher announce the event ID to connected dashboards. If the announcement is missed, the stored row remains available for backfill.

A useful record includes a stable event ID, device ID, status, observation time, and an optional reason. The ID needs a total order within the scope being read. That scope might be one operator, one game region, or one fleet partition. A global sequence is easier to reason about but can concentrate writes; partitioned sequences distribute work but make the client keep several cursors. The dashboard's authorization boundary and query pattern should decide.

Keep the live payload small. An event ID, or an ID plus enough fields for an optimistic render, makes the contract clear: receipt is useful but non-authoritative. The client fetches the canonical representation and advances its durable cursor only after applying the event.

This is where a notebook prototype can conceal the hard part. A callback appending messages to an in-memory list demonstrates motion, not recovery. The production evaluation must interrupt the connection between commit, publish, receive, render, and cursor persistence. Each interruption exposes a different duplicate-or-gap case.

The pattern is explained most clearly by its cost. Durable events add storage, indexes, retention work, authorization checks on replay, and extra reads whenever a client catches up. It is not suitable for disposable signals such as a pointer moving across a lobby screen, where the next position replaces the last and replay has no user value. Partitioned cursors also complicate clients, while one global cursor can become an awkward write boundary. Those are real trade-offs. Use the inbox only for notifications whose loss would violate the user promise; keep replaceable presence and animation state on the lighter realtime path. That separation reduces the amount of history the team must retain and evaluate without pretending every publish has reliable delivery semantics.

Reconnect without opening a race

On initial load, the dashboard reads ordered pages after cursor zero, renders each event idempotently by event ID, and records the highest contiguous applied ID. It then attaches to the live channel. There is a race between finishing the read and attaching, so the client reads once more from its recorded cursor after the subscription is active. That final catch-up closes the gap.

Reconnect uses the same path. A live announcement for event 421 should trigger a catch-up read rather than blindly move the cursor to 421, because events 419 and 420 may be absent locally. The ordered store decides what was missed.

Short rule: never advance across a hole.

Gaps win.

The client also needs bounded pages and a policy for expired history. If its cursor is older than retained data, an empty response would silently corrupt the view. The application should explicitly signal that incremental recovery is unavailable, rebuild from an authorized snapshot, and continue from the snapshot's cursor. Retention is a product requirement, not a transport setting.

Different cursors express different facts. A server-side delivered cursor says the backend attempted delivery. A per-device applied cursor supports independent recovery on a laptop and wall display. A per-user read marker captures human acknowledgement. Calling all three last-seen invites accidental data loss.

Failure injection before deployment

Happy-path latency is the least revealing measurement. An evaluation harness should generate a known sequence of device transitions, record the expected IDs, and interrupt the dashboard at controlled points. Then it compares the final rendered set and order with the durable record.

The minimum useful cases are concrete:

disconnect before an event is committed;
disconnect after commit but before announcement;
receive the same announcement twice;
reconnect while new events are being written;
resume from cursors just inside and just outside retention.

For every case, assert three invariants: no missing authorized event, no duplicated user-visible notification, and a cursor that never advances beyond the highest contiguous applied ID. The third catches a subtle failure. If IDs 418 and 420 arrive, persisting 420 erases evidence that 419 is missing.

Measure catch-up duration, unread-event count, duplicate announcements, backfill page count, and the age of the oldest unrecovered event. Also measure storage and read amplification. Frequent heartbeats can inflate writes; huge pages can reduce request count while increasing memory and time to first recovered item. Optimize from the recovery distribution, not the always-connected demo.

If an AI component summarizes operator notifications, keep it outside the delivery-critical path. Store the source event first, run enrichment asynchronously, and retain the original status when enrichment is late. An eval set should verify that summaries preserve device ID, status, and observation time. Token use and enrichment latency matter, but neither should decide whether the base alert exists.

Match durability to the user promise

An inbox is not required for every pixel that moves. Ephemeral cursor motion in a multiplayer lobby may be discarded because newer state supersedes it. A device-offline alert assigned to an operator has different semantics: durable identity, authorization, retention, and read state. Combining both classes under one reliability promise either wastes resources or loses history.

Use a current-state snapshot when only the latest value matters. Use an ordered inbox when transitions matter to a human workflow. Use both when the dashboard needs a fast fleet overview and an auditable alert timeline. This semantic choice comes before scaling.

Before copying the pattern, define the recovery objective in observable terms: maximum backfill time, retention interval, ordering scope, and whether internal duplicate delivery is acceptable when the UI remains idempotent. Then test those claims under churn. A live channel remains valuable, but it should carry urgency rather than sole custody of the notification.