ESSA: Evolutionary Strategies for Scalable Alignment

LLM RLHF LoRA/Adapter ML EA
大规模语言模型(LLMs)越来越多地依赖对齐技术,以确保其输出符合人类偏好。尽管基于人类反馈的强化学习(RLHF)是当前主流方法,但它在计算资源、内存需求和训练稳定性方面存在较高成本,尤其是在模型规模扩大时更为明显。本文提出了一种新的对齐框架 ESSA(Evolutionary Strategies for Scalable Alignment),该框架采用进化策略(Evolutionary Strategies,ES)来高效地对齐 LLM,而无需进行梯度计算。 进化策略因其良好的特性,非常适合用于 LLM 的对齐任务,这些特性包括高度可并行化、内存效率高、对稀疏奖励具有鲁棒性,以及从强大的预训练策略出发时所需收敛的数据样本更少。此外,进化策略减少了对大量超参数调优的需求,使对齐过程更加简单且稳定。虽然进化策略在低维优化任务中表现优异,但在应用于高维的 LLM 时却面临挑战。 为应对这一挑战,我们提出了一种参数高效的架构改进方案,通过低秩适应(low-rank adaptation)降低优化问题的维度。我们在数学推理任务上使用可验证的、基于准确率的指标对我们的方法进行了评估,结果表明,ESSA 比基于梯度的方法(如 Group Relative Policy Optimization,GRPO)收敛更快,数据利用效率更高。我们的研究结果表明,进化策略是一种有前景且可扩展的梯度对齐替代方案,为高效地进行大语言模型的后训练打开了新的可能性。
Large Language Models (LLMs) are increasingly relying on alignment techniques to ensure that their outputs match human preferences. Although reinforcement learning from human feedback (RLHF) is the dominant approach, it has high computational costs, memory requirements, and training instability, particularly when scaling to larger models. This paper introduces ESSA (Evolutionary Strategies for Scalable Alignment), a new framework that uses Evolutionary Strategies (ES) to efficiently align LLMs without the need for gradient computation. ES is well-suited for LLM alignment due to its favorable properties, such as high parallelizability, memory efficiency, robustness to sparse rewards, and fewer data samples required for convergence, especially when starting from a strong pre-trained policy. Moreover, ES eliminates the need for extensive hyperparameter tuning, making the alignment process simpler and more stable. Although ES excels in low-dimensional optimization, it poses a challenge when applied to high-dimensional LLMs. To address this challenge, we propose a parameter-efficient architectural modification that reduces the dimensionality of optimization through low-rank adaptation. We evaluated our approach on mathematical reasoning tasks with verifiable accuracy-based metrics, demonstrating that ESSA converges faster and is more data efficient than gradient-based methods like Group Relative Policy Optimization (GRPO). Our findings establish ES as a promising and scalable alternative to gradient-based alignment, paving the way for efficient post-training of large language models.
许愿