bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix

LLM RLHF ML RL POA
强化学习(RL)对语言模型的后训练阶段,优化器中始终维护全精度(fp32)的主权重,并从一个经“就近舍入”(round-to-nearest)转换得到的 bfloat16(bf16)副本中提供策略 rollout。在当前常用的优化学习率下,绝大多数单步权重更新量均小于 bf16 量化间隔的一半,因此该舍入操作会直接丢弃这些微小更新:以 Qwen3-1.7B 模型为例,每步仅有约 0.4–0.5% 的服务中权重实际发生变化;而生成训练数据所依赖的策略模型,其参数状态比优化器滞后 50–150 步。现有各类不匹配校正方法均无法察觉这一滞后现象,因其比较的是同一轮舍入操作之后产生的两个副本,二者共享相同的舍入误差源。 为此,我们提出“公平投射”(Fair Cast)方法:在导出主权重时采用无偏舍入规则(即随机舍入或误差反馈),并直接基于服务中使用的 bf16 副本计算训练损失。该改动仅需在权重导出调用处增加一行代码,无需引入任何新超参数,且每次导出 1.7B 参数仅额外耗时 22–38 毫秒。在 Qwen3 系列模型(参数量覆盖 0.6B 至 4B)、分别搭配 AdamW 和 Muon 优化器的共计 25 组严格配对的随机种子实验中,Fair Cast 在 18 组中表现更优、6 组持平、仅 1 组略逊,平均提升达 6.1 分;在斑马谜题(zebra puzzles)任务上,Fair Cast 四次运行结果为 0.57 ± 0.16,而采用就近舍入的方法则坍缩至 0.33 ± 0.10。此外,Fair Cast 还使得保存的 fp32 检查点与实际服务中的 bf16 模型完全一致;更重要的是,它将“就近舍入”所隐含的参数滞后显式揭示为一种可调控的正则化效应——当特定任务需要时,这种滞后效应亦可被有意识地恢复与利用。
Reinforcement learning (RL) post-training of language models keeps fp32 master weights in the optimizer and serves rollouts from a bf16 copy produced by round-to-nearest conversion. At the learning rates in common use, most per-step updates are smaller than half a bf16 spacing and the conversion discards them: on Qwen3-1.7B only 0.4-0.5% of served weights change per step, and the policy that generates the training data lags the optimizer by 50-150 steps. Existing mismatch corrections cannot see this lag because they compare two copies downstream of the same rounding. We propose Fair Cast: export the master with an unbiased rounding rule (stochastic rounding or error feedback) and compute the training loss on the served copy. The change is one line at the export call, adds no hyperparameters and costs 22-38 ms per export of 1.7B parameters. Across 25 matched seed-pairs on Qwen3-0.6B to 4B with AdamW and Muon, Fair Cast is ahead on 18, tied on 6 and behind on 1, with a mean gain of 6.1 points; on zebra puzzles it reaches 0.57 +/- 0.16 over four runs where round-to-nearest collapses to 0.33 +/- 0.10. It also makes the saved fp32 checkpoint agree with the served model, and it exposes the implicit lag of round-to-nearest as a regularizer that can be restored explicitly when a task needs it.
许愿