Base Models Can Reason By Taking a Cue From Training Data

LLM RLHF CoT AI Safety / AI Ethics AVAR
本文中,我们研究了训练数据如何在基础模型响应开头的词元(tokens)与后续的推理行为之间建立关联。 首先,我们证明:若固定某些特定的起始词元提示(cues),基础模型在数学与编程任务上的表现即可媲美其经强化学习(RL)训练后的对应版本。例如,“.\n\nOkay”这一提示可将Olmo-3-7B模型在MATH-500基准上的pass@1准确率从42%提升至78%;而“Alright,”则可使Qwen3-14B模型的相应准确率从72%升至87%。 其次,强化学习训练会显著提高这些提示词元出现的概率;而若人为固定使用这些提示,则能在很大程度上复现强化学习相对于基础模型所获得的性能增益。 第三,我们将不同词元提示所引发的推理效应溯源至训练数据本身。我们通过因果性数据干预实验,成功将任意一个普通词汇(如“chicken”)转化为有效的推理提示,或消除某一既有提示的实际效果。类似的数据编辑操作甚至能使提示指令“Think duck duck goose”(“思考鸭子、鸭子、鹅”)在激发推理能力方面与经典提示“Think step by step”(“逐步思考”)同样有效。此外,我们还发现:不同提示所诱导出的隐状态表征,与训练语料中不同类型文档的分布呈现出显著相关性。 最后,我们进一步拓展对词元提示的研究,以大语言模型的安全性为案例展开分析,发现不同提示会诱发截然不同的拒绝响应与服从行为,而这些行为模式恰恰分别对应于训练数据中不同类型的内容(例如安全对齐数据与常规指令遵循数据)。
In this paper, we study how training data creates associations between the tokens at the start of a base model's response and the reasoning behavior that follows. First, we demonstrate that fixing particular starting token cues makes a base model's performance competitive with that of its reinforcement learning (RL)-trained counterparts on math and coding. For instance, the cue ".\n\nOkay" raises Olmo-3-7B's MATH-500 pass@1 accuracy from 42% to 78%, while "Alright," raises Qwen3-14B's from 72% to 87%. Second, RL makes these cues more likely, while fixing them recovers much of its performance gain over the base model. Third, we trace the reasoning effects of token cues to the training data. We perform causal data interventions to turn an arbitrary word, such as "chicken", into an effective reasoning cue, or remove an existing cue's effect. A similar edit makes the prompt instruction "Think duck duck goose" as effective as "Think step by step" at eliciting reasoning. We also find that the hidden state representations induced by different cues correlate with different document types from the training set. Finally, we extend our study of token cues with a case study in language model safety, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.
许愿