TailSFT: Filtered Fine-Tuning Improves Post-Training Performance

LLM SFT RLHF ML RL FT
强化学习的后训练过程正推动现代人工智能系统在推理与智能体能力方面的进步;然而,越来越多的研究表明,只有当强化学习被用于微调一个本身已具备较强能力的基础模型时,其效果才最为显著。我们由此提出疑问:当前主流的模型训练流程所产出的模型,是否真的最适合开展后续的强化学习?基于先前研究——该研究指出“覆盖度”(coverage)与“pass@K”指标可作为预测强化学习后性能的重要依据——我们设计了一种对监督微调(SFT)的简易改进方法,称为TailSFT:它在训练过程中主动过滤掉那些模型已能较好拟合的样本序列,从而将学习重心转向数据分布中建模不足的区域,即分布的“尾部”(tail)。我们通过受控实验与理论分析相结合的方式,为TailSFT的设计选择(尤其是所采用的具体过滤标准)提供了充分的合理性论证与实证验证。在OLMo-3 7B模型上,TailSFT在数学与编程类评测任务的pass@16指标上普遍取得提升,绝对增益最高达17%,且仅带来极小的额外计算开销。更重要的是,这些具有更高覆盖度的中间检查点,在后续采用GRPO算法进行强化学习时,稳定地转化为最高达4个百分点的pass@1绝对提升,充分证明TailSFT产出的检查点更适合作为强化学习的初始化起点。此外,我们还提出了一种轻量级诊断工具,用于快速识别TailSFT最可能发挥效用的训练场景。更广泛而言,本研究的结果启示我们应采取一种原则性强、阶段意识明确的模型研发范式:即在模型开发的每个中间阶段,其产出的检查点不应仅以自身性能为评判标准,而更应以其对后续训练阶段的支持效能为根本衡量依据。
Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.
许愿