RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation

AK's Picks Embodied AI and Robotics Imitation Learning SRT World Models GenAI Diffusion TDIG
2026年07月07日
规模化机器人学习需要海量且多样化的轨迹数据,而当前数据采集严重受限于物理遥操作——每一次示范都必须占用操作员的时间,并严格绑定于特定的硬件设备和作业空间。为此,我们提出“数字遥操作”(digital teleoperation)这一新范式,通过以生成式世界模型替代真实机器人,从而彻底解除数据采集对物理条件的依赖。在该框架下,操作员连续输出的手部姿态流驱动一个以机器人为中心的生成式世界模型,仅凭单张参考图像即可合成高保真的第一人称视角视频。所记录的姿态流作为与具身形态无关的动作标签,可通过标准动作重定向技术无缝迁移至任意目标机器人,由此生成完整的状态-动作轨迹,且完全不依赖于任何物理硬件,可直接用于模仿学习。我们将该范式具体实现为 RynnWorld-Teleop 系统,其整合了深度感知的骨骼条件建模、面向视频的扩散变换器(Video Diffusion Transformer)上从人体到机器人的渐进式训练,以及流式自回归知识蒸馏技术。该流水线将生成过程压缩为单次前向推理,可在单块 H100 GPU 上实现每秒 40 帧以上的实时交互式生成。仅使用 RynnWorld-Teleop 生成的数据训练的策略,即可在灵巧、多样的双手协同任务中实现高效的“仿真到现实”(Sim2Real)零样本迁移。此外,将本系统生成的数字遥操作数据作为增强数据加入真实世界数据集后,各项任务的成功率均得到持续提升,充分证明 RynnWorld-Teleop 是一种高保真、可大规模扩展的数据引擎,有望赋能下一代机器人智能体的发展。
Scaling robot learning requires massive, diverse trajectory data, yet collection is currently bottlenecked by physical teleoperation, where every demonstration binds operator time to specific hardware and workspaces. We introduce digital teleoperation, a paradigm that decouples data collection from physical constraints by replacing the real robot with a generative world model. In this framework, an operator's hand-pose stream drives a robot-centric generative world model to synthesize high-fidelity egocentric videos from a single reference image. The recorded pose stream serves as an embodiment-agnostic action label transferable to any target robot via standard retargeting, yielding complete state-action trajectories for imitation learning independent of physical hardware. We instantiate this paradigm in RynnWorld-Teleop, a system that integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer, and streaming autoregressive distillation. This pipeline compresses the generative process into a single-pass inference, enabling 40+ FPS, real-time interactive generation on a single H100 GPU. Policies trained exclusively on RynnWorld-Teleop-generated data achieve effective zero-shot Sim2Real transfer across dexterous and diverse bimanual tasks. Moreover, augmenting real-world datasets with our digitally teleoperated data consistently improves success rates, demonstrating that RynnWorld-Teleop serves as a high-fidelity, scalable data engine for the next generation of robotic agents.
许愿