RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

LLM SFT RLHF ML RL PGPO
1527
2026年09月29日
强化学习中奖励可验证（RLVR）的常规范式是让智能体对任务进行多次尝试，并以成功案例为目标进行优化。然而，这一范式在自改进（self-improvement）场景中会面临严峻挑战：此时任务难度极高，智能体几乎无法成功（成功率极低甚至为零），且不存在教师模型或可供蒸馏的示例解法。本文提出一种新方法——RLTL;DR。具体而言，在每次失败尝试后，我们向策略模型展示验证器（verifier）的输出结果，并引导其自主撰写一条简明扼要的反馈，即一条“TL;DR”式的核心洞见（insight）。后续的推理过程（rollout）则以此前所有积累的洞见为条件，并持续采样推理直至找到正确解。此外，我们进一步支持对上下文中的这些洞见进行反向传播（backpropagation），从而促使模型将“任务→洞见”的映射关系内化为自身能力。我们在若干极具挑战性的工具调用（tool-calling）与编程数据集上进行了评测（这些数据集经筛选后，标准方法在Pass@128指标下性能为0%）；采用标准GRPO算法训练Qwen 3.5 9B“思考型”策略模型时，其Pass@1指标始终停滞于0%–1%之间；而RLTL;DR则成功突破了这一学习瓶颈：在训练阶段将洞见置于上下文中时，Pass@1达到14%–31%；尤为关键的是，在评估阶段完全不提供任何洞见（即零上下文）时，仍能稳定取得12%–13%的Pass@1成绩。我们发现，性能跃升的关键正在于“任务→洞见”映射关系的有效内化。为进一步探究该机制，我们将方法简化为SFTL;DR：仅基于（任务，洞见）二元组进行监督微调（SFT），完全不呈现也不对任何推理过程进行反向传播。实验表明，仅需使用4,000条此类二元组即可恢复几乎全部RLTL;DR及传统基于完整推理轨迹的SFT训练所达到的性能。这揭示了一种极具前景的精简型训练范式，其核心思想可概括为：“面对此类任务，应牢记此类要点”。我们期望这一思路能为未来研究带来启发。
The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.
许愿