Preference Alignment with Flow Matching

我们提出了Preference Flow Matching（PFM），这是一种新的基于偏好的强化学习（PbRL）框架，可以简化偏好与任意类别的预训练模型的集成。现有的PbRL方法需要对预先训练的模型进行微调，这带来了可扩展性、效率和需要模型修改等挑战，特别是在像GPT-4这样的黑盒API中。相比之下，PFM利用流匹配技术直接从偏好数据中学习，从而减少对预先训练模型的大量微调的依赖。通过利用基于流的模型，PFM将不太受欢迎的数据转化为受欢迎的结果，并有效地将模型输出与人类偏好对齐，而不依赖于明确或隐含的奖励函数估计，从而避免了奖励模型中的常见问题，如过度拟合。我们提供了理论洞见，支持我们的方法与标准PbRL目标的对齐。实验结果表明，我们的方法在实践中具有实际效果，为将预训练模型与偏好对齐提供了新的方向。

Preference-based reinforcement learning (PbRL) methods require extensive fine-tuning of pre-trained models, which presents scalability, inefficiency, and model modification challenges. The paper aims to streamline the integration of preferences into an arbitrary class of pre-trained models, reducing the dependency on fine-tuning and avoiding issues like overfitting in reward models.

The paper proposes Preference Flow Matching (PFM), a framework that directly learns from preference data using flow matching techniques. By transforming less preferred data into preferred outcomes, PFM aligns model outputs with human preferences without relying on explicit or implicit reward function estimation. The method leverages flow-based models and offers a new direction in aligning pre-trained models to preference.

The paper provides theoretical insights supporting PFM's alignment with standard PbRL objectives and experimental results indicating the practical effectiveness of the method. The authors used various datasets to evaluate PFM's performance and compared it to existing PbRL methods. The paper also discusses the limitations of the proposed method and suggests future research directions. The code for PFM is available on GitHub.

Related work includes various PbRL methods that rely on fine-tuning pre-trained models, such as COACH and PEARL. Other methods, such as DeepRL and Inverse Reinforcement Learning, require explicit or implicit reward function estimation. The paper also discusses flow-based models, which have been used in generative modeling and density estimation.