跳到正文
DEV Community· Quill Datafix·· 1 小時前AI 評分54

如何用 30 分鐘檢查 SAC 實作是否正常

The 30-minute sanity check for an RL implementation

AI 導讀

Quill Datafix 建議在執行昂貴強化學習任務前,先用 Pendulum-v1 對 SAC 實作訓練 100k steps,並每 10k steps 檢查確定性策略的評估回報。

正文

Quill Datafix

Numbers from one CPU-only box (16 cores), PyTorch 2.15+cpu, a from-scratch ~300-line SAC, seed 0. Every number comes from the eval.csv logs shipped in the repo below.

Draft for dev.to. Numbers from one CPU-only box (16 cores), PyTorch 2.15+cpu,
my own ~300-line SAC, seed 0. Filled from runs/Pendulum-v1_s0/eval.csv.

Before you trust any RL implementation on an expensive task, run it on a cheap one
where failure is loud. On continuous control that task is still Pendulum-v1, and it
costs pocket change on CPU.

The test

  1. Train your algorithm on Pendulum-v1 for 100k steps with your standard hyperparameters.
  2. Watch the deterministic-policy evaluation every 10k steps.
  3. Read the shape, not just the endpoint.

What healthy looks like

Measured on my implementation, seed 0, one gradient step per environment step
(runs/Pendulum-v1_s0/eval.csv):

steps eval return (10 episodes, deterministic)
10k -123 ± 57
20k -120 ± 53
50k -148 ± 52
70k -197 ± 98
80k -216 ± 71
100k -168 ± 78

That is the honest shape of a healthy but noisy run: it converges into the
healthy regime early and then oscillates inside it. the fix is not more steps of
the same thing; it is the standard tweak — two gradient steps per environment
step — which trades ~2x wall clock for a better final return (-111 vs -168
here) at comparable variance. (Both runs ship in the package so you can
compare the two curves yourself.)

How to read failures

  • Flat around -1,000: the policy gradient is not flowing — check that you are backpropagating through the sampling (rsample), not .sample().
  • Improves then collapses to -1,000: the entropy temperature collapsed to zero and the policy froze; check target_entropy and the alpha loss sign.
  • Noisy but trending to -150 ± 20: fine. Pendulum's practical target is about -150; scores near -130 are already at the ceiling of this task.
  • NaNs: tanh correction log(1 - tanh²) needs an epsilon; without it the log blows up at saturated actions.

Why Pendulum and not CartPole

CartPole is discrete; it will not test your squashed-Gaussian machinery, which is
where almost every SAC bug lives. Pendulum is continuous, badly scaled, and its
optimal return is comfortably below zero — so an implementation that "looks
trained" because numbers are increasing can still be broken, and an
implementation that converges near -150 has exercised the whole pipeline:
rsample, tanh correction, twin critics, target Polyak averaging, and learned
temperature.

The 30-minute budget

With 256x256 networks, batch 256 and one gradient step per environment step, the
full 100k-step Pendulum run took about 14 minutes on 16 CPU cores (838 s, wall
clock, measured). There is no excuse for skipping this check before a GPU run.


The SAC implementation and all logs referenced here ship in
https://gitlab.com/quilldatafix/sac-mujoco-repro; every number above is reproducible with one command.


Code, training logs and checkpoints (MIT): sac-mujoco-repro · Ready-to-run packages: Datafix Tools

來源:DEV Community · dev.to