Test-Time Scaling Makes Overtraining Compute-Optimal

Nicholas Roberts ,

Sungjun Cho ,

Zhiqi Gao ,

Tzu-Heng Huang ,

Albert Wu ,

Gabriel Orlanski ,

Avi Trost ,

Kelly Buchanan ,

Aws Albarghouthi ,

Frederic Sala

AK's Picks LLM PTSAD MQIA

现代大语言模型（LLM）在推理阶段具备可扩展性，例如通过重复采样（repeated sampling）提升性能，但其推理开销会随模型规模及采样次数同步增长。这一特性引出了一个权衡问题，而传统的预训练缩放定律（如Chinchilla定律）并未对此加以考虑。本文提出“训—推联合缩放定律”（Train-to-Test Scaling Laws，简称 $T^2$），在端到端总预算固定的前提下，对模型参数量、预训练数据量（以token计）以及推理阶段采样次数这三个关键维度进行联合优化。$T^2$ 在传统预训练缩放定律基础上进行了现代化升级：一方面引入面向推理时缩放的 pass@$k$ 建模方法，另一方面将预训练决策与推理时决策纳入统一优化框架。$T^2$ 的预测结果在不同建模路径下均展现出稳健性：既可通过任务损失（task loss）量化三者联合缩放的整体效应，也可通过任务准确率（task accuracy）刻画其对最终性能的影响。我们在八个下游任务上开展实证分析发现：当将推理开销纳入考量后，最优预训练策略发生根本性转变——模型需大幅“过训练”（overtraining），其对应的预训练规模远超标准预训练缩放定律（如Chinchilla）所建议的常规范围。我们进一步通过实验验证该结论：在 $T^2$ 预测出的最优过训练区域中实际预训练若干高度过训练的模型，结果证实其性能显著优于仅依据传统预训练缩放定律所确定的模型。最后，鉴于当前前沿大语言模型普遍经历后训练（post-training）阶段，我们还验证了上述结论在后训练之后依然成立，从而表明 $T^2$ 缩放定律对现代大模型的实际部署具有切实指导意义。

Modern LLMs scale at test-time, e.g. via repeated sampling, where inference cost grows with model size and the number of samples. This creates a trade-off that pretraining scaling laws, such as Chinchilla, do not address. We present Train-to-Test ($T^2$) scaling laws that jointly optimize model size, training tokens, and number of inference samples under fixed end-to-end budgets. $T^2$ modernizes pretraining scaling laws with pass@$k$ modeling used for test-time scaling, then jointly optimizes pretraining and test-time decisions. Forecasts from $T^2$ are robust over distinct modeling approaches: measuring joint scaling effect on the task loss and modeling impact on task accuracy. Across eight downstream tasks, we find that when accounting for inference cost, optimal pretraining decisions shift radically into the overtraining regime, well-outside of the range of standard pretraining scaling suites. We validate our results by pretraining heavily overtrained models in the optimal region that $T^2$ scaling forecasts, confirming their substantially stronger performance compared to pretraining scaling alone. Finally, as frontier LLMs are post-trained, we show that our findings survive the post-training stage, making $T^2$ scaling meaningful in modern deployments.

许愿