強化學習獎勵塑形令混合語言模型推理軌跡縮短最多 65%
Reward shaping slashes reasoning steps by 65%
相關研究指，為混合語言模型調整強化學習獎勵可重新啟動其 THINK 推理模式，令推理軌跡平均縮短最多 65%，並大致維持答案準確度。在 9B 參數設定中，生成限制內完成率由 45.9% 升至 93.9%，平均軌跡長度由 2,494 降至 868 tokens。測試只涵蓋 Qwen3 系列與 StartLux-Decision 及各自的基準；準確度仍有輕微下降，能否延伸至其他架構或任務尚不清楚。
Tailored reinforcement‑learning rewards reactivate the dormant THINK circuitry in hybrid language models, chopping chain‑of‑thought traces by roughly two‑thirds without sacrificing answer quality. The surprise is that a modest reshaping of the reward signal does more than steer behavior—it resurrects an internal reasoning mode that the model already possessed.
Before these studies, practitioners either forced models to emit full CoT sequences or relied on static heuristics that never touched the model’s latent THINK pathway. RL was popular for improving performance in the NO‑THINK regime, but its gains were credited to newly learned policies rather than to any reclamation of existing reasoning capacity.
RLCD² lifts the share of reasoning traces that finish within the generation limit by up to 48 percentage points and slashes average trace length by as much as 65 % while largely preserving decision accuracy. At 9B parameters, termination climbs from 45.9 % to 93.9 %, and mean trace length drops from 2,494 to 868 tokens—a full 65 % reduction [1].
The think win rate surges from 50 % to 94 % for the 8B checkpoint, with comparable jumps to 86 % for 4B and 84 % for 1.7B, meaning that after RL training a response is judged more THINK‑like than an original NO‑THINK output in the overwhelming majority of pairwise comparisons [2].
“Original-to-RL patching of the think subspace lowers accuracy to 0.244, removing 75 % of the RL gain, and response length falls from 7.0k to 4.2k tokens while reflection words fall from 13.0 to 3.9 per response” – an experiment that directly ties the reward‑induced performance boost to a low‑dimensional THINK‑related activation subspace [2].
These results are confined to specific hybrid architectures (the Qwen3 series and StartLux‑Decision) and their respective benchmark suites—arithmetic and tool‑use evaluations for the Qwen3 experiments and decision benchmarks such as Decision Index and DecisionPivot for StartLux‑Decision—and it remains unclear whether the same reward shaping will transfer to pure decoder‑only models or to other task domains. Moreover, the reported “largely preserving accuracy” still masks modest drops that could matter for high‑stakes applications, suggesting a trade‑off curve that has not been fully mapped.
If the 65 % reduction in reasoning steps holds across deployments, developers can slash inference latency and token‑processing costs by swapping in the publicly released LoRA adapters for the Qwen3 models (provided in the think‑leakage repository) that implement the reward‑shaping objective, without retraining the base model. This cheap post‑hoc tweak turns expensive chain‑of‑thought pipelines into leaner, faster decision engines.
References
來源：DEV Community · dev.to