When Does Muon Help Agentic Reinforcement Learning?

Kai Ruan ,

Jinghao Lin ,

Zihe Huang ,

Ziqi Zhou ,

Qianshan Wei ,

Xuan Wang ,

Hao Sun

AK's Picks LLM RLHF Agent Agent Planning ML RL POA

缪子优化器（Muon）在大规模预训练中可与AdamW相媲美，但其在强化学习（RL）后训练阶段的价值尚不明确。我们通过在ALFWorld环境中、以Qwen2.5-0.5B-Instruct模型为基础，采用单种子严格匹配的对比实验，研究了标准版Muon在稀疏奖励智能体强化学习中的表现。在“组内组策略优化”（GiGPO）框架下，仅将Muon应用于隐藏层权重矩阵，即可使最终验证窗口的成功率从0.290提升至0.546（增幅达88%）；而采用高学习率的AdamW基线方法，在完成更新后则完全无法取得任何成功。该提升效果依赖于优势函数估计器（advantage estimator）的选择与学习率的设定：当学习率为3e-5时，Muon将GRPO的成功率从0.161提升至0.268；而在GraphGPO中，其晚期验证窗口的成功率差距则在接近饱和状态时显著收窄。当学习率降至1e-5时，采用Muon的GraphGPO进一步将成功率提升至0.901，将归一化验证AUC值从0.399提高至0.556，并分别提前30轮和60轮迭代达到0.5和0.75的成功率阈值。这些探索性结果表明，Muon确实有望提升智能体强化学习的性能，也提示我们有必要将策略优化器、优势函数估计器与学习率三者作为一个整体进行协同研究。多随机种子验证及跨任务泛化能力评估仍有待后续开展。

Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.

许愿