跳到正文
DEV Community· Viswaretas Kotra·· 29 分鐘前AI 評分52

FSDP 與 DDP:何時需要分片數據並行?

FSDP vs DDP: when do you need sharded data parallel?

AI 導讀

DDP 在每張 GPU 保留完整模型狀態,以較低通訊量換取較高顯存佔用;FSDP 分片模型狀態以節省顯存,但增加通訊量。按每參數約 16 bytes 的模型狀態估算,80 GB H100 預留 20 GB 給激活值時,DDP 的上限約為 3.75B 參數。

正文

For full training with AdamW in mixed precision, DDP stops fitting on an 80 GB NVIDIA H100 at roughly 3.75 billion parameters once you keep 20 GB for activations, and above that you need FSDP. Nodus runs either one with the same torchrun command on 1, 2, 4 or 8 GPUs of a machine, with H100s listed at $2.60/hr on the Nodus pricing page. If one GPU holds the full copy of weights, gradients and optimizer state with room for activations, DDP is simpler and moves less data; if not, FSDP shards that copy across GPUs.

Our multi-GPU training guide covers writing and launching both. This post only settles the choice between them. It works out per-GPU memory for 1B, 8B, 13B and 70B models, the traffic each strategy adds, and the smallest setup that fits. All figures are our own arithmetic from the PyTorch docs and the ZeRO paper, shown so you can redo them.

What is the actual difference between DDP and FSDP?

DDP copies the whole model to every GPU, and FSDP splits it. The DDP docs describe synchronizing gradients across each model replica with an all-reduce, after which every GPU runs the same optimizer step on its own full copy.

The FSDP2 fully_shard docs say FSDP "shards module parameters, gradients, and optimizer states across data parallel workers to save memory at the cost of communication." Before each layer runs, FSDP all-gathers that layer's full weights, uses them, and frees them again. After backward, it reduce-scatters the gradients so each GPU keeps only its slice and updates only its slice of the optimizer state.

That last clause is the whole trade. DDP pays memory to save traffic. FSDP pays traffic to save memory.

How much memory does each strategy need per GPU?

Full training with Adam in mixed precision takes about 16 bytes per parameter before activations. The ZeRO paper counts 2 bytes for 16-bit weights, 2 for 16-bit gradients and 12 for the fp32 master weights plus Adam's two moments. Plain torch.autocast with fp32 weights lands on the same 16: 4 for weights, 4 for gradients, 8 for the moments.

Each strategy divides those 16 bytes differently across N GPUs:

  • DDP keeps all 16 bytes on every GPU.
  • FSDP FULL_SHARD keeps 16/N bytes per GPU, plus the one layer it is gathering at the moment.
  • SHARD_GRAD_OP keeps the full 16-bit weights for the whole forward and backward pass, then shards everything else: about 2 + 14/N bytes. The FSDP docs describe it as unsharding before forward and only resharding after backward.

Here is that worked for 8 GPUs, which is one full H100 machine:

Model DDP SHARD_GRAD_OP on 8 FULL_SHARD on 8 Left on an 80 GB H100 with FULL_SHARD
1B 16 GB 3.75 GB 2 GB 78 GB
8B 128 GB 30 GB 16 GB 64 GB
13B 208 GB 48.75 GB 26 GB 54 GB
70B 1,120 GB 262.5 GB 140 GB does not fit

The arithmetic is parameters x 16 for DDP, parameters x (2 + 14/8) for SHARD_GRAD_OP and parameters x 16 / 8 for FULL_SHARD, in decimal GB. Activations, all-gather buffers and the CUDA context come on top, so read these as floors.

Per-GPU model state for full training with AdamW on 8 GPUs: 1B takes 16 GB with DDP, 3.75 GB with SHARD_GRAD_OP and 2 GB with FULL_SHARD; 8B takes 128, 30 and 16 GB; 13B takes 208, 48.75 and 26 GB.
Only the 1B model fits under DDP; both FSDP strategies hold the 8B and 13B models well inside 80 GB.

So DDP works for the 1B model and nothing larger in this table. The 8B and 13B models fit on one 8 x H100 machine under either FSDP strategy. A full 70B run needs 140 GB per GPU even fully sharded over 8, which no 80 GB GPU can hold.

Where does the 3.75 billion parameter line come from?

It is 60 GB divided by 16 bytes per parameter. We keep 20 GB of each 80 GB GPU free for activations, which is a planning assumption, not a measurement. Long sequences or large micro-batches need more, and activation checkpointing needs less.

With no reserve at all, the ceiling is 80 / 16 = 5 billion parameters. Real runs sit somewhere between the two, so a 3B model is a DDP job and a 7B model is an FSDP job on H100s. For anything near the line, measure one step's peak with the snippet in our CUDA out of memory guide before you choose.

What does FSDP cost you in communication?

About 1.5 times DDP's traffic for FULL_SHARD, and about the same as DDP for SHARD_GRAD_OP. The ZeRO paper puts DDP's all-reduce at 2 times the parameter count per step, since an all-reduce is a reduce-scatter plus an all-gather. Fully sharded training adds one more all-gather and comes to 3 times.

The FSDP docs show where the extra pass comes from. FULL_SHARD gathers each layer's weights before forward, frees them, and gathers them again before backward. SHARD_GRAD_OP keeps them from forward to backward, so it skips the second gather and costs the memory you saw in the table.

FSDP is built to hide this traffic by fetching the next layer's weights while the current one runs. That works best on a fast link such as NVLink inside one machine. We have not benchmarked the gap, and it depends on your model, batch size and interconnect, so we won't quote a throughput number. Across machines it matters more, which is why hybrid sharding exists.

Which FSDP sharding strategy should you pick?

Start with FULL_SHARD, the default, and move only when the numbers tell you to.

Strategy Per-GPU state Traffic per step Use it when
DDP 16 bytes per param 2 x params The full copy fits with room for activations
SHARD_GRAD_OP about 2 + 14/N bytes 2 x params FSDP is needed, the 16-bit weights fit, and you want less traffic
FULL_SHARD 16/N bytes 3 x params Memory is tight, or the model is large
HYBRID_SHARD 16 / GPUs per machine 3 x params inside a machine, gradient all-reduce across You train on several machines and one machine's GPUs can hold the sharded state

HYBRID_SHARD, called HSDP in FSDP2, applies FULL_SHARD within a machine and replicates across machines, per the FSDP docs. Its catch is that adding machines never lowers per-GPU memory. A 70B model still needs 140 GB per GPU with 8 GPUs per machine, however many machines you add.

In FSDP2 these are arguments rather than enum names. The fully_shard docs map them like this:

from torch.distributed.device_mesh import init_device_mesh
from torch.distributed.fsdp import fully_shard

# Pick ONE of the following.

# FULL_SHARD: the default for every block except the root
for block in model.layers:
    fully_shard(block)
fully_shard(model)

# SHARD_GRAD_OP: keep gathered weights from forward to backward
for block in model.layers:
    fully_shard(block, reshard_after_forward=False)
fully_shard(model, reshard_after_forward=False)

# HYBRID_SHARD (HSDP): replicate across 2 machines, shard across 8 GPUs in each
mesh = init_device_mesh("cuda", (2, 8))
for block in model.layers:
    fully_shard(block, mesh=mesh)
fully_shard(model, mesh=mesh)

A 2D mesh tells fully_shard to replicate on the first dimension and shard on the second. Create the optimizer after sharding so it sees the sharded parameters.

What is the smallest H100 setup for each model size?

Fewer GPUs means a smaller hourly bill, though not a cheaper job. Using the same 20 GB activation reserve and the per-machine GPU counts Nodus offers (1, 2, 4 or 8, per the Nodus GPU guide), the smallest single-machine setups are:

Model Strategy GPUs Model state per GPU Price per hour on Nodus
1B DDP (or no wrapper at all) 1 x H100 16 GB $2.60
8B FULL_SHARD 4 x H100 32 GB $10.40
13B FULL_SHARD 4 x H100 52 GB $10.40
70B FULL_SHARD 8 x B200 (180 GB) 140 GB $44.00

H100 SXM 80 GB is listed at $2.60/hr and B200 180 GB at $5.50/hr on the Nodus pricing page. Prices checked November 19, 2026. The 8B model fails our reserve on 2 GPUs (64 GB each), and the 13B model with SHARD_GRAD_OP needs 8 GPUs (48.75 GB each).

Hourly price of the smallest setup that fits full AdamW training with 20 GB left for activations: 1B on 1 x H100 at $2.60, 8B on 4 x H100 at $10.40, 13B on 4 x H100 at $10.40, and 70B on 8 x B200 at $44.00.
FULL_SHARD puts an 8B or 13B full fine-tune on half an H100 machine; 70B needs bigger GPUs, not more of them.

The total bill follows GPU-hours, not GPU count. A job that needs 80 H100-hours costs $208.00 at $2.60 whether it runs on 4 GPUs for 20 hours or 8 GPUs for 10, if it scales evenly across them. Measure that for your own model before assuming it.

For 70B, the larger GPU is the better path. An 8 x H200 machine at 141 GB per GPU leaves 1 GB after 140 GB of state, so it does not work. Our 70B VRAM guide goes through that model in more detail, including the LoRA and QLoRA options.

Where is DDP the better choice?

DDP wins whenever the model fits, and it is not a beginner's fallback. It moves 2 times the parameters per step instead of 3, and every GPU holds a complete model, so saving a checkpoint is one state_dict() from rank 0.

Gradient accumulation is cheaper too. DDP's no_sync() skips the gradient all-reduce on accumulation steps. FSDP2's set_requires_gradient_sync(False) skips gradient reduction too, but with FULL_SHARD the weights are still gathered for every micro-batch's forward and backward.

There are also fewer ways to get it wrong. FSDP needs a wrapping plan, since sharding only the root gathers the whole model at once. It also needs sharded or gathered checkpoints and clipping that accounts for split gradients; FSDP1 has its own clip_grad_norm_() for this. With DDP, none of this applies.

Before switching to FSDP for a model that almost fits, try the cheaper fixes. The DDP docs say gradient_as_bucket_view=True saves memory equal to the gradients' size, and activation checkpointing can free several GB. If you are fine-tuning, LoRA shrinks gradients and optimizer state so much that most models fit with plain DDP.

How do you run DDP or FSDP on Nodus?

Use the same command for both, because the choice lives in your script. The Nodus GPU guide runs a DDP smoke test on every GPU of one machine with:

nodus run --name torchrun-multi-gpu --gpu A6000:2 --max-cost 1.00 -- torchrun train.py

Change A6000:2 to H100:4 or H100:8 for the setups above. Nodus sets PET_NPROC_PER_NODE to the GPU count, so torchrun starts one process per GPU without flags, and /dev/shm is already sized for NCCL. --max-cost is the dollar cap for the job.

Across machines, add --nodes and --launcher torchrun as shown in the multi-node guide, which is in Beta. That guide says a two-machine DDP result does not qualify cross-machine FSDP, so check its list of exercised combinations before a large sharded run over several machines. One machine of bigger GPUs, as in the 70B row, avoids the question.

FAQ

Is FSDP slower than DDP?

It moves more data: 3 times the parameter count per step with FULL_SHARD, against 2 times for DDP. Whether that shows up as lower throughput depends on how much of it overlaps with compute on your interconnect, so measure both if the model fits either way. On Nodus, a sweep runs one Job across parameter values and compares cost, time and throughput per cell, which suits a DDP vs FSDP test.

Is FSDP the same as DeepSpeed ZeRO-3?

They follow the same idea from the ZeRO paper. FULL_SHARD shards weights, gradients and optimizer state like ZeRO stage 3, and SHARD_GRAD_OP keeps whole weights during compute, which is close to stage 2.

Should new code use FSDP1 or FSDP2?

FSDP2, the fully_shard API. Its docs say it shards each parameter on dim-0 as a DTensor, keeps throughput similar to FSDP1, and allows sharded state dicts without all-gathers. PyTorch publishes a migration guide for FSDP1 users.

Does FSDP help with inference?

Not much. FSDP's savings come mostly from gradients and optimizer state, which inference does not have. To serve a model too big for one GPU, use a serving engine with tensor parallelism instead.

Originally published on the Nodus blog.

來源:DEV Community · dev.to