Dex-One2Many: Learning Dexterous Manipulation from a Single Human Demonstration

Embodied AI and Robotics Imitation Learning SRT ML RL
18
2026年10月08日
尽管从单段人类操作视频中学习灵巧操作，为替代成本高昂的机器人示范提供了一条颇具前景的新路径，但近期许多方法主要局限于对示范动作的直接模仿。这种严格的动作匹配往往严重制约了策略在初始物体位姿、目标位姿及抓取方式等未在视频中出现场景下的泛化能力。相比之下，通过强化学习（RL）来自主发现控制策略虽具备更广泛的泛化潜力，但在缺乏先验引导的情况下，面对复杂多阶段任务中高维状态-动作空间的探索，其学习效率极低、成功率有限。为协同解决上述泛化性与探索效率这一对耦合挑战，我们提出了Dex-One2Many——一种“真实→仿真→真实”的端到端框架，能够仅凭单段人类操作视频，学习得到具有强泛化能力的灵巧操作策略。我们的核心洞见在于：将输入视频抽象为一系列场景图（scene graphs），并以此作为强化学习的结构化引导信号，在保障高效探索的同时，维持策略的广泛泛化性。这些场景图一方面充当生成式约束条件，用于采样多样化的环境重置状态（reset states）；另一方面为每个操作阶段提供稠密奖励信号（dense rewards）。由于场景图所约束的是物体之间、物体与手部之间的相对关系（如“工具握持于手掌中”“刀刃朝向食材”），而非具体的空间位姿，因此所生成的重置状态自然覆盖了远超原始视频范围的物体位姿与抓取方式；而在此类多样化状态下，借助各阶段对应的稠密奖励进行初始化训练，则可显著缩短探索路径，并使探索过程始终受到明确引导。Dex-One2Many全程仅在仿真环境中完成训练，即可实现零样本迁移（zero-shot transfer），直接部署于真实的多指灵巧手平台。在涵盖五类典型工具使用与灵巧操作任务的实验中，Dex-One2Many在“已见配置”（seen configurations）下性能较基线方法提升6.5%；而在更具挑战性的“未见场景”（unseen scenarios）下，其鲁棒泛化能力进一步凸显，性能优势扩大至71%。
While learning dexterous manipulation from a single human video offers a promising alternative to costly robot demonstrations, many recent methods predominantly imitate demonstrated motions. Such strict motion matching often limits generalization to initial object poses, goal poses, and grasps not shown in the video. Alternatively, discovering a policy via reinforcement learning (RL) allows for broad generalization, but without prior guidance, it struggles with high-dimensional exploration in complex, multi-stage tasks. To address these coupled generalization and exploration challenges, we present Dex-One2Many, a real-to-sim-to-real framework that learns a generalizable dexterous manipulation policy from a single human video. Our key insight is to abstract the video into sequential scene graphs that guide RL, enabling efficient exploration while preserving broad generalizability. The graphs serve as generative constraints for sampling diverse reset states and provide dense rewards for each stage. Because the graphs constrain relations rather than exact poses, these reset states cover object poses and grasps beyond the video, while initializing each stage from them with dense rewards keeps exploration short and guided. Trained entirely in simulation, Dex-One2Many transfers zero-shot to a real multi-fingered hand. Across five tool-use and manipulation tasks, Dex-One2Many exceeds baselines by 6.5% in seen configurations, while its robust generalization widens this gap to 71% in unseen scenarios.
许愿