Copy-Paste Language Models

LLM PTSAD Long Context
1332
2026年10月13日
人们通常直观地认为，Transformer 模型在从输入中进行内容复制方面具有较高效率。事实上，我们证明：通过显式建模，大语言模型（LLM）在上下文中的复制能力可以得到显著提升。为此，我们提出了“复制-粘贴语言模型”（Copy-Paste Language Models, CPLMs），即在传统语言建模（LM）头之外，额外引入一个轻量级的“复制-粘贴”头。CPLMs 不仅提升了训练效率与下游任务性能，还在长上下文建模和检索增强场景中提供了全新的性能-开销权衡点（operating points）。我们通过预训练参数量从 1.35 亿到 17 亿不等的多个模型，全面验证了该方法的有效性——结果表明，无论模型规模或主干网络（backbone）如何变化，CPLMs 均能带来一致的性能提升，且训练与推理开销几乎可以忽略不计。我们进一步揭示了 CPLMs 成功的内在机制：它通过提供一条替代性的梯度通路，在一定程度上缓解了传统语言建模头所面临的梯度瓶颈问题。
Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.
许愿