RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

AK's Picks Embodied AI and Robotics Imitation Learning World Models Robot Morphologies GenAI Diffusion CV Monocular or Binocular Depth Estimation and Occlusion Handling
2026年07月07日
在开放世界中实现机器人操控,不仅需要识别场景的外观,还需预判其三维结构在交互作用下的运动方式。我们认为,同步采集的RGB图像、深度图与光流场(即RGB-DF)构成了一种具有物理基础的表征形式,能够准确刻画场景内在的四维动态特性。相较于二维像素视频,这种多模态协同机制将视觉外观、几何结构与时间运动三者对齐,所构建的表征空间显著贴近机器人系统底层执行器(end-effector)所需的动作信号,从而有效弥合了“世界预测”与“策略学习”之间的鸿沟。基于这一洞见,我们提出了RynnWorld-4D——一种生成式模型,它能在单一统一的扩散过程中,以单帧RGB-D图像及一条自然语言指令为输入,联合生成未来若干帧的RGB图像、深度图与光流场。该四维世界模型采用三支路架构,融合跨模态注意力机制与逐帧三维旋转位置编码(3D RoPE),确保外观、几何与运动三类信息在演化过程中保持高度一致性。为支撑大规模训练,我们构建了Rynn4DDataset 1.0数据集:这是一个海量数据集,涵盖超过2.544亿帧第一人称视角下的人类与机器人操控视频,并为其中每一帧提供了高质量的深度图与光流场伪标签。此外,我们进一步提出RynnWorld-4D-Policy——一种逆动力学头部模块,它可直接利用RynnWorld-4D内部生成的四维表征,在单次前向传播中完成推理,无需耗时的多步去噪过程,即可闭环输出机器人控制动作。实验结果表明:RynnWorld-4D能生成在时间与空间维度上均高度连贯的四维预测;而RynnWorld-4D-Policy在真实世界的灵巧双臂操控任务中达到当前最优性能,尤其在对空间精度与时间协调性要求严苛的任务中表现尤为突出。
Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow, namely RGB-DF, provide a physically grounded representation that captures the underlying 4D dynamics of a scene. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to the low-level end-effector actions demanded by robotic systems, thereby narrowing the gap between world prediction and policy learning. Building on this insight, we introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate Rynn4DDataset 1.0, a massive dataset of over 254.4 million frames across egocentric human and robotic manipulation videos with high-quality pseudo-labels for depth and optical flow. We further propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of RynnWorld-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks, particularly excelling in tasks demanding spatial precision and temporal coordination.
许愿