Freeform Preference Learning for Robotic Manipulation

Embodied AI and Robotics Imitation Learning RC ML RL Inverse RL
69
2026年06月30日
奖励函数设计仍是提升自主机器人策略性能的核心瓶颈，尤其在长时程操作任务中：稀疏的成功标签所提供的学习信号过于匮乏，而二元偏好则将多种相互竞争的质量维度强行压缩为单一且模糊的信号。为此，我们提出了“自由形式偏好学习”（Freeform Preference Learning, FPL）方法，旨在从人类以自然语言表达的自由形式偏好中学习机器人策略。与传统方式——即要求标注者简单判断两条轨迹中哪一条整体更优——不同，FPL允许标注者自主定义自然语言描述的偏好维度（如运动速度、安全性、放置质量、操作谨慎程度等），并针对每一维度分别提供成对轨迹的相对偏好判断。这些标注数据被用于训练一个语言条件化的奖励模型，该模型能根据输入的轨迹及对应的偏好维度标签，输出该维度上的具体奖励值。随后，我们利用该模型训练一个奖励条件化策略，使其能够协同优化多个由人类指定的偏好维度。在四项真实世界任务及两项仿真环境中的长时程操作任务上，FPL相较稀疏奖励方法和二元偏好方法平均提升了38个百分点。除性能显著提升外，FPL还能在不依赖显式子任务划分的前提下，自动学习到稠密的渐进式学习信号；所学策略展现出超越训练数据的行为组合性（compositional behavior）；更重要的是，用户可在测试阶段仅通过调整偏好维度指令，即可实时引导策略表现出不同行为模式，而无需重新训练模型。相关博客文章及演示视频请见：https://freeform-pl.github.io/fpl.website/
Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/
许愿