跳到正文
Google Developers Blog·· 1 小時前AI 評分52

跨都會區大語言模型訓練:MaxText 中的 Streaming DiLoCo

Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText

AI 導讀

Google 在 MaxText 實作 Streaming DiLoCo,將模型參數分片並錯開同步,以避免每隔 H 步傳送完整模型造成的週期性停頓。

正文

NOV. 19, 2026

Chris Zuo Software Engineering Intern

Introduction

Traditionally, training state-of-the-art Large Language Models (LLMs) relies on a monolithic cluster of accelerators, where all devices are collocated and tightly coupled via high-bandwidth to constantly synchronize gradients. However, physical and cost constraints make scaling a single slice increasingly difficult.

To scale further, we can connect multiple compute slices across geographically distributed datacenters. However, when training across multiple datacenters over wide-area networks (WAN), standard data-parallel training suffers from severe network stalls because slow WAN transfers cannot be fully overlapped with computation.

To address this, the Google DeepMind and Google Research teams developed DiLoCo (Distributed Low-Communication), a distributed optimization algorithm designed to make training resilient to these low-bandwidth, high-latency connections. In this blog post, we discuss the core concepts of DiLoCo, how it is implemented in MaxText (Google's open-source JAX-based LLM training framework), and the performance benefits it brings to large-scale LLM training on Google Cloud.

What is DiLoCo?

DiLoCo was introduced by Google DeepMind in the paper DiLoCo: Distributed Low-Communication Training of Language Models. Instead of synchronizing gradients across all devices at every single training step (as in standard Data Parallelism), DiLoCo partitions training across M independent compute groups, referred to as learners, and allows each learner to perform many local optimization steps independently before communicating across groups.

The Inner-Outer Optimization Loop

DiLoCo operates on two levels:

  1. The Inner Loop (Local): Each learner (typically a single TPU slice or an intra-datacenter multi-slice group) runs a standard optimizer (typically AdamW) on its local data partition for H steps (the synchronization cadence, e.g., H = 37–65 in our experiments). During these steps, no communication occurs between different DiLoCo groups. Standard intra-replica communication (such as ICI, or intra-replica DCN if the learner itself is a multi-slice group) continues at every step as usual.
  2. The Outer Loop (Global): Every H steps, the M learners compute the difference (delta) between their current weights and the global weights at the start of the interval. The learners average these deltas across the network via a distributed all-reduce and apply an outer optimizer (typically SGD with Nesterov momentum) to update the global weights, which then serve as the starting point for the next inner-loop interval.

Despite communicating significantly less frequently (by a factor of H, e.g., 1-2 orders of magnitude less) than standard synchronous training, DiLoCo has been shown to achieve comparable convergence and model quality.

The Vanilla DiLoCo described above reduces total communication volume by a factor of H, however, all learners still transmit their weight deltas for the entire model all at once at every H-th step. On a thin network pipe, that full-model transfer creates a large periodic stall every H steps. This issue is alleviated by Streaming DiLoCo, which is described in the next section.

Why Streaming DiLoCo Shines in Low-Bandwidth Regimes

In multi-slice training, communication within a single slice is handled by high-speed Inter-Chip Interconnects (ICI). When scaling to multiple slices, synchronization across slices must traverse the Data Center Network (DCN) or, if the slices are geographically distributed, wide-area networks (WAN).

Within a single datacenter, DCN bandwidth is high enough that JAX and XLA can often hide standard Data Parallel all-reduce collectives by overlapping gradient communication with backward-pass computation. Across datacenters or congested links, however, bandwidth drops and latency rises sharply. Standard DP stalls at every step waiting for global gradient synchronization, collapsing Model FLOPS Utilization (MFU).

Streaming DiLoCo eliminates this synchronization stall by partitioning the model's parameters into P disjoint fragments. Over the course of H steps, every single fragment in the model is synchronized exactly once, achieving the exact same outer-loop cadence as vanilla DiLoCo without spiky bandwidth utilization.

  • Fragmentation: MaxText partitions the model into P = NLAYERS + 1 fragments: one fragment for all non-scanned parameters (token embeddings, final layer norm, and output head) and Nlayers fragments, one for each scanned Transformer decoder layer.
  • Staggered Per-Step Streaming: Instead of pausing every H steps to exchange the entire model, MaxText synchronizes a single fragment every Δh=max⁡(1,round(H/P)) steps. Setting H=P (Δh=1) synchronizes one fragment per step, spreading the exact same reduced communication volume across all H steps so the network sees a nearly uniform, tiny stream (once large embedding matrices are load-balanced across the layer fragments, as described next) rather than a massive periodic spike. Rather than synchronizing adjacent layers consecutively, the selection of which fragment syncs at each interval is interleaved across the depth of the network (e.g., Fragment 0, then Layer k, then Layer 2k, etc.). This ensures every round of synchronization touches different depths of the network rather than leaving an entire section of the model stale while another section updates.
  • Embedding Load Balancing: Since Fragment 0 holds all parameters outside the scanned layers: the token embeddings, pre-logits layer norm, and unembedding/output head, its size depends on the vocabulary dimension, and it can be much larger than the other fragments. Without mitigation, the step that synchronizes Fragment 0 would produce a massive spike in network traffic compared to the other layer fragments. To this end, while some non-scanned vectors remain in Fragment 0, large matrices from this fragment are evenly split and merged with the other Nlayers fragments.

Sizing Your Network Link: Data Parallel vs. Vanilla vs. Streaming DiLoCo

Using a roofline model (similar to the JAX Scaling Book), we can calculate the network bandwidth required to prevent communication stalls for a model with ∣Θ∣ parameters (at b bytes per parameter, e.g., b=2 for bfloat16) and per-step compute time tstep=tfwd+tbwd. For example, the standard Data Parallel algorithm transfers 2∣Θ∣b bytes per sync via an all-reduce collective, which is a distributed reduce-scatter where each node reduces a different part, followed by an all-gather so all participants receive all parts. To achieve zero communication stalls, we overlap the reduce-scatter of the gradients with the backward pass, which produces them and takes tbwd time. Therefore, the peak required bandwidth is Bdp ≈ 2∣Θ∣b / tbwd.

Table 1 (2)

Worked Example (Qwen3-32B in bfloat16, tstep​≈6.2s, P=H=65):

  • Data Parallel exchanges ~65.5 GB (∼524 Gb) of gradients at every single step. To finish this transfer within a ∼6s step time without stalling, it requires at least ∼85 Gbps of sustained cross-slice bandwidth (524 Gb / 6.2 s≈85 Gbps). On a constrained, inter-slice 10 Gbps link, transferring 65.5 GB would take over 50 seconds, causing massive stalls.
  • Vanilla DiLoCo (H=65) runs at full speed for 64 local steps and reduces total traffic by 65x (∼65.5 GB over 65 steps), but still stalls on every 65th step while transferring the full ∼65.5 GB payload at once
  • Streaming DiLoCo (P=H=65) exchanges just 1/65th of the model (∼1.0 GB) per step (∼65.5 GB over 65 steps—65× less traffic than DP), requiring only ∼1.3 Gbps of bandwidth. Even on a heavily throttled 10 Gbps link, communication fits comfortably inside the step time.

Evaluating the Claims: Results and Insights

1. Performance and Network Resilience

To evaluate distributed efficiency under realistic network bottlenecks, we benchmarked Qwen3-32B (64 decoder layers, partitioned into P = 65 fragments with H = 65) across 2× TPU v5p-32 slices (dcn_diloco_parallelism=2) using built-in Linux traffic-control (tc) bandwidth shaping to limit inter-slice bandwidth across three network regimes: Unthrottled (Line Rate), 30 Gbit/s aggregate (Moderate Congestion), and 10 Gbit/s aggregate (Severe Congestion).

Table 2 - updated

image3

Figure 1: Cross-Slice DCN Communication Volume (Qwen3-32B on 2× v5p-32 over 65 steps). By synchronizing one parameter fragment (1/65th of the model) per step, Streaming DiLoCo reduces cross-slice DCN traffic from 4,258.8 GB (Data Parallel baseline) to 67.0 GB—a 64× to 65× reduction (∼1.0 GB/step vs. 65.5 GB/step).

image1

Figure 2: Mean step time across network congestion regimes on 2× v5p-32. Under 30 Gbit/s and 10 Gbit/s inter-slice throttling, Streaming DiLoCo achieves a 3.37× and 9.57× speedup (−89.5% step latency at 10 Gbit/s, Δ = 49.35s) over Data Parallel (5.76 s/step vs. 19.41 s/step and 55.11 s/step) and outperforms Vanilla DiLoCo (6.39 s/step and 7.55 s/step) by eliminating periodic full-model synchronization stalls.

Key Findings:

  • Line Rate to Moderate Congestion (Unthrottled to 30 Gbit/s): When inter-slice bandwidth is unthrottled, XLA overlaps DP gradient reductions with backward computation, and all three methods achieve similar step times (~5.6–5.9 s). At 30 Gbit/s inter-slice bandwidth, however, Data Parallel already saturates the link and degrades to 19.41 s/step, while Streaming DiLoCo maintains a flat 5.76 s/step (a 3.37× speedup).
  • Severe Network Constraints (10 Gbit/s): Data Parallel degrades sharply from 5.58 s to 55.11 s/step as its 65.5 GB per-step all-reduce saturates the link. Vanilla DiLoCo (H = 65) fares much better (7.55 s/step, a 7.30× speedup), but still suffers a periodic full-model synchronization bubble every 65 steps. Streaming DiLoCo (P = 65, H = 65) maintains a flat 5.76 s/step—delivering a 9.57× speedup over DP and eliminating the step-H stall.
  • Favorable Scaling with Depth and Batch Size: Because we set P = N_layers + 1 (one decoder layer per fragment), scaling model depth increases per-step compute time across all layers while Streaming DiLoCo's per-step communication stays divided by P, giving the system even more compute time per step to amortize background weight transfers. Importantly, scaling P (and therefore H) with model depth does not degrade convergence: in our experiments on dense Qwen3-32B (64 decoder layers, P = H = 65), Streaming DiLoCo matches the synchronous Data Parallel baseline loss with no degradation in model quality.

2. Convergence and Model Quality

A common question with low-communication training is whether allowing learners to train independently between synchronizations degrades model quality—especially on Mixture-of-Experts (MoE) architectures, where independent updates could cause expert routers to drift apart.

We evaluated Streaming DiLoCo against synchronous Data Parallel pre-training across two MoE architectures in MaxText (Qwen3-30B-A3B and Gemma4-26B-A4B) over 4,000 training steps:

Table 3 - updated

image4

Figure 3: Pre-training loss trajectory comparing Streaming DiLoCo against synchronous Data Parallel on Qwen3-30B-A3B over 4,000 steps. After brief early-warmup synchronization adjustments in the first ~800 steps, Streaming DiLoCo smoothly tracks and slightly outperforms the Data Parallel baseline

Empirical results confirm that Streaming DiLoCo preserves—and even slightly improves—validation loss:

  • Outer Nesterov Momentum: Rather than naively averaging learner weights, stepping an outer Nesterov momentum optimizer (β = 0.9, outer learning rate η_outer between 0.3 and 0.7) across fragment pseudo-gradients stabilizes the global trajectory.
  • Warm Inner Optimizers: Because synchronization updates only the model parameters while leaving each learner's local AdamW first- and second-moment buffers intact, learners never lose optimizer momentum after a fragment sync.

Getting Started with Streaming DiLoCo in MaxText

MaxText provides a native JAX and Flax NNX implementation of both Vanilla and Streaming DiLoCo—requiring no external central parameter servers.

1. Configuration

You can enable Streaming DiLoCo in your MaxText YAML configuration or CLI overrides by setting enable_diloco: true and enable_streaming_diloco: true. As a best practice, set both num_diloco_fragments (P) and diloco_sync_period (H) to N_layers + 1 so that exactly one fragment is synchronized per training step (Δh = 1):

# Enable DiLoCo and Streaming DiLoCo
enable_diloco: true
enable_streaming_diloco: true

# Partition into N_layers + 1 fragments (e.g., 36 layers + 1 embedding/head = 37 for Qwen3-8B,
# 48 + 1 = 49 for Qwen3-30B-A3B, or 64 + 1 = 65 for Qwen3-32B)
num_diloco_fragments: 37
diloco_sync_period: 37
use_sequential_layers: false  # false = interleaved layers across fragments (recommended)

# Outer Nesterov SGD optimizer hyperparameters
diloco_outer_lr: 0.7
diloco_outer_momentum: 0.9

# Multi-slice topology: number of DiLoCo learner islands across DCN/WAN
dcn_diloco_parallelism: 2  # e.g., 2 TPU slices (or set to -1 to infer from num_slices)

# Optional: emulate constrained WAN bandwidth on fast clusters via Linux tc
# dcn_bandwidth_limit: "10gbit"

Plain text

Copied

2. Under the Hood & Further Reading

In MaxText, diloco.py splits each global batch across the M learners so each learner processes GBS / M tokens per step. To overlap cross-slice communication with TPU computation in Streaming DiLoCo, weight deltas for the scheduled fragment are offloaded to M-way sharded syncer workers running on the learner host CPUs. Rather than routing through a single central coordinator, the host-CPU syncer shards execute a distributed all-reduce across the network in the background to average the fragment deltas while the TPUs continue running inner-loop training steps. Each syncer shard then applies the outer Nesterov momentum update locally and writes the updated fragment weights back to its colocated TPU learner. To dive deeper or launch your first run:


Conclusion

Streaming DiLoCo in MaxText decouples large-scale LLM pre-training from the requirement of a single, monolithic, high-bandwidth fabric. By combining use of an inner and outer optimizer with fine-grained parameter streaming, engineering teams can pool TPU slices across congested datacenter networks or geographically distributed metros—sustaining high hardware utilization without compromising model convergence. Streaming DiLoCo is available today in MaxText v0.2.5 on PyPI, the corresponding Docker images and on GitHub main. Try it today for experimenting with distributed training workloads.

Previous

Next

來源:Google Developers Blog · developers.googleblog.com