Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learning

LLM RLHF Long Context ML RL DREM
2026年07月21日
能够生成分步推理过程的大型语言模型在复杂任务上已展现出卓越性能,而将其扩展至长上下文场景已成为当前一项重要前沿课题。然而,我们在该场景下发现了一种关键性失效模式:即“重复性复制”——模型大量照搬输入文本内容至其推理过程之中,而非积极、有效地解决问题。我们证实,此类行为在当前主流的长上下文大语言模型中普遍存在,且随着上下文长度增加而愈发严重。通过将每个提示(prompt)拆解为与任务相关的核心证据和无关的干扰性上下文,我们进一步揭示出其根本原因在于“接地不足”(insufficient grounding):模型不加区分地从提示中复制内容,而那些未能聚焦于核心证据的模型,其回答错误的概率显著更高。受此诊断结果启发,我们提出了GEAR(Grounding Evidence-Aware Reward,即“证据感知型接地奖励”)方法——一种奖励塑形技术:它在原有基于准确率的奖励信号基础上,额外引入两项新信号:一是“接地奖励”,衡量模型输出与核心证据之间的重合度;二是“干扰项惩罚”,对模型输出与无关上下文之间的重合予以扣分。为使GEAR适用于自然语言数据,我们开发了一套自动化流程,可从任意文档中自动构建带有证据标注的训练数据。我们在多个模型规模和基准测试集上对GEAR进行了系统验证,结果表明:相较于仅依赖准确率的传统强化学习方法,GEAR带来了一致且显著的性能提升,平均提升幅度达+4.6分;尤其在更长上下文场景下增益更为明显;同时,该方法还有效减少了重复性复制现象及冗长低效的推理链长度。我们的研究发现表明:即便长上下文评估的重点正从简单的信息检索逐步转向复杂的多步推理,模型能否精准锚定并利用相关证据,仍是一项不可或缺的核心能力,且目前仍有巨大的改进空间。
Large language models that generate step-by-step reasoning traces have achieved strong performance on complex tasks, and extending them to long-context settings has emerged as an important frontier. However, we identify a critical failure mode in this regime: \emph{repetitive copying}, where models extensively copy text from the input into their reasoning traces rather than productively solving the problem. We show that this behavior is pervasive across frontier long-context LLMs and intensifies with context length. By separating each prompt into task-relevant key evidence and irrelevant distractor context, we further show that the root cause is insufficient grounding: models copy from the prompt indiscriminately, and those that fail to focus on key evidence are far more likely to answer incorrectly. Motivated by this diagnosis, we propose GEAR (Grounding Evidence-Aware Reward), a reward shaping method that augments the accuracy signal with a grounding reward for overlap with key evidence and a distractor penalty for overlap with irrelevant context. To enable GEAR on natural-language data, we develop an automated pipeline that constructs evidence-annotated training data from arbitrary documents. We validate GEAR across multiple model scales and benchmarks, showing consistent improvements of up to +4.6 average points over standard RL with accuracy-based rewards, with larger gains at longer contexts, while also reducing repetitive copying and thinking length. Our findings suggest that, even as long-context evaluation shifts from simple retrieval toward complex reasoning, accurate grounding in relevant evidence remains an indispensable capability with substantial room for improvement.
许愿