Predicting the Order of Upcoming Tokens Improves Language Modeling

AK's Picks LLM PTSAD ML LM
491
2025年08月26日
多标记预测（MTP）作为一种辅助目标被提出，用于改进语言模型训练中的下一项预测（NTP），但其提升效果并不一致，在标准自然语言处理（NLP）基准测试中表现欠佳。我们认为，MTP对未来的标记进行精确预测作为辅助损失函数过于困难。因此，我们提出了标记顺序预测（Token Order Prediction，简称TOP），该方法通过使用学习排序（learning-to-rank）损失函数，训练模型根据即将出现的标记与其当前位置的接近程度对其进行排序。与MTP所需的多个Transformer层相比，TOP仅需额外增加一个反嵌入层。我们使用NTP、MTP和TOP目标对参数量分别为3.4亿、18亿和70亿的模型进行了预训练。在八个标准NLP基准测试中的结果显示，即使在模型规模扩大的情况下，TOP整体上仍优于NTP和MTP。我们的代码可在以下地址获取：https://github.com/zaydzuhri/token-order-prediction
Multi-Token Prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, underperforming in standard NLP benchmarks. We argue that MTP's exact future token prediction is too difficult as an auxiliary loss. Instead, we propose Token Order Prediction (TOP), which trains models to order upcoming tokens by their proximity using a learning-to-rank loss. TOP requires only a single additional unembedding layer compared to MTP's multiple transformer layers. We pretrain models of 340M, 1.8B, and 7B parameters using NTP, MTP, and TOP objectives. Results on eight standard NLP benchmarks show that TOP overall outperforms both NTP and MTP even at scale. Our code is available at https://github.com/zaydzuhri/token-order-prediction
许愿