Robots Acquire Manipulation Skills in Seconds from a Single Human Video

Embodied AI and Robotics Imitation Learning IL
2026年07月22日
机器人需具备快速、轻松地习得新技能,同时牢固保持已掌握技能的能力。然而,当前主流方法仍依赖于繁琐的训练时循环,不仅成本高昂、耗时漫长,还会导致已有技能的退化。本文提出 HOST(面向机器人的单样本人类技能习得框架),使机器人仅凭一段人类操作视频,即可在数秒内完成新技能的学习,且完全保留此前已掌握的各项技能。HOST 通过一种“自 grounding 的级联预测”机制实现技能习得:首先,估计机器人在所演示任务中的当前进展;继而,将后续任务进程映射为机器人自身在未来时刻可能观测到的状态;最终,基于这些预测的未来观测结果,推导出具体执行动作。该级联预测模型的训练目标由视频演示直接驱动——我们先将机器人实际轨迹与人类视频演示共同映射至一个共享的“任务进展流形”(task progress manifold),再据此重新定义每个训练目标,使其严格对齐视频中展现的未来任务演进过程。由此,HOST 赋予机器人主动跟随人类演示流程的能力,并能将其灵活适配至机器人自身的物理构型与运动特性。在推理阶段,HOST 仅需一段人类视频,平均耗时 29 秒即可习得一项新技能,平均成功率高达 62%;相较零样本迁移基线方法,成功率提升达 45%,且全程不损害既有技能;甚至在性能上超越了每项任务均需使用 50 段机器人实操演示进行微调的基线方法——而 HOST 所需的演示数据量仅为后者的 1/50,单个技能的习得速度更加快达 507 倍。有关 HOST 的更多详情,请访问项目官网。
The ability to acquire skills rapidly and effortlessly while retaining those already mastered is essential for robots. However, current methods still rely on a cumbersome training-time loop that is costly and slow, while eroding skills already mastered. In this paper, we introduce HOST (Human-to-robot One-Shot Skill AcquisiTion), a framework that enables a robot to acquire skills in seconds from a single human video while retaining previously mastered skills. HOST resolves skill acquisition through a cascade of self-grounded prediction. It first estimates the robot's progress within the demonstrated task, then translates the upcoming progression into the robot's own future observations, and finally derives actions from these predicted observations. This cascade is trained on targets coupled to the video demonstration, obtained by mapping the robot trajectory and the video demonstration onto a shared task progress manifold, then redefining each target to align with the future progression of the video. HOST thereby enables the robot to actively follow the demonstrated procedure and adapt it to the robot's embodiment. HOST acquires novel skills at inference time from a single human video in an average of 29 seconds and achieves a 62% average success rate. It exceeds the zero-shot baseline by 45% while retaining previously mastered skills. HOST even exceeds the baseline fine-tuned on 50 robot demonstrations per task while requiring 50 times fewer demonstrations and acquiring each skill 507 times faster. Additional information about HOST is available on the project website.
许愿