Milvus 如何透過遠端 TransformLog 訂閲及 MVCC 屏障改善分佈式串流

Enhancing Distributed Streaming in Milvus: Remote TransformLog Subscriptions and MVCC Barriers

Distributed systems that rely on streaming replication face a delicate balance between efficiency, consistency, and fault tolerance. Milvus, a popular vector database, recently tackled a subtle but critical issue in its streaming replication mechanism. By introducing remote TransformLog subscriptions with ordered, resumable delivery and multi-version concurrency control (MVCC) barriers, Milvus now supports more reliable recovery and consistent query progress tracking.

In this article, we'll explore the problem Milvus faced, the technical mechanisms introduced in the fix, and the broader lessons for anyone implementing distributed streaming with strong consistency guarantees.

The Problem: Inefficient Recovery and Inconsistent Streaming Progress

Before this change, Milvus’s streaming replication lacked support for remote TransformLog subscriptions that could deliver entries in order and resume from interruptions. This gap led to two main issues:

• Inefficient recovery: Without resumable cursors and ordered delivery, replaying logs during recovery was unreliable and potentially redundant.

• Inconsistent query progress tracking: The system did not preserve MVCC barriers properly, which are essential for snapshot isolation during queries. This caused checkpoints to advance prematurely and queries to see inconsistent or stale data.

Additionally, the backpressure mechanism was simplistic — it relied on a scanner mode rather than actual cache occupancy, which failed to regulate load effectively and maintain stable delivery rates.

These issues combined to create a streaming replication system that was fragile under failure conditions and complicated to maintain strong consistency guarantees.

The Fix: Remote Subscriptions, MVCC Barriers, and Cache-Aware Backpressure

Remote TransformLog Subscriptions with Resumable Cursors

The core innovation was introducing a new SubscribeTransform RPC mechanism that supports both local and remote owner routing. This subscription model includes a handshake protocol to establish and close subscriptions gracefully. Key features include:

• Ordered Entry and SyncUp delivery: Ensuring that TransformLog entries arrive in the correct sequence, which is crucial for consistent replay and state updates.

• Resumable cursors: If a connection drops or is interrupted, the subscription can resume from the last known position, avoiding redundant processing and speeding up recovery.

• Backoff and reconnect strategies: Failed subscription attempts back off exponentially, and reconnects are suspended when there are no active subscriptions, conserving resources.

• Idle stream reclamation: Streams that are no longer in use are cleaned up promptly, preventing resource leaks.

Aligning Recovery with WAL Consumer

Recovery logic was aligned with the Write-Ahead Log (WAL) Consumer by filtering replay events before they reach data-only handlers. This ensures that replayed events are consistent and that recovery proceeds in a controlled manner.

MVCC Barrier Coordination

MVCC barriers are preserved and scoped progress is published in WAL order without advancing durable checkpoints prematurely. This means:

• Volatile barriers are queued until RecoveryStorage confirms prefix completeness: This prevents queries from seeing inconsistent snapshots.

• Durable checkpoints advance only after safe conditions: Avoiding premature checkpoint advancement maintains snapshot isolation and query correctness.

Cache-Aware Backpressure

The previous backpressure mechanism based on scanner mode was replaced with a more sophisticated cache-occupancy-based admission control:

• WAB cache uses capacity and TTL eviction: This ensures that the cache does not overflow and that stale entries are evicted timely.

• RecoveryStorage consumption constraints: These constraints help regulate admission based on actual resource usage.

• Configurable slowdown and rejection thresholds: When the cache occupancy crosses certain thresholds, the system slows down or rejects new entries to maintain stability.

Additional Improvements

• WALSummary reads reuse verified chunk footers: This optimization prevents repopulation of retired entries, improving efficiency.

• Support for independent reader cancellation and GC snapshot protection: This enhances the robustness of concurrent reads and garbage collection.

Takeaway: Combining Resumable Cursors, MVCC Barriers, and Backpressure for Strong Consistency

Implementing distributed streaming subscriptions with strong consistency is challenging. Milvus’s approach highlights several key principles:

• Resumable cursors and handshake protocols are essential for robust, ordered, and fault-tolerant delivery.

• MVCC barriers must be carefully coordinated to preserve snapshot isolation and prevent premature checkpoint advancement.

• Backpressure should be cache-aware and resource-sensitive, not simplistic or mode-based, to maintain stability under load.

Together, these mechanisms enable efficient recovery, consistent query progress tracking, and reliable streaming replication.

For engineers building distributed streaming systems, this case study underscores the importance of integrating consistency controls with resource management and connection resilience.

This article is based on the merged PR in the milvus-io/milvus repository titled "feat: add remote TransformLog subscriptions and MVCC barriers." Read the full breakdown on nossletter.tech.

Originally published on Nossletter.