VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Yen-Jen Wang ,

Jiaman Li ,

Sirui Chen ,

Takara E. Truong ,

Pei Xu ,

Pieter Abbeel ,

Rocky Duan ,

Koushil Sreenath ,

Angjoo Kanazawa ,

Carmelo Sferrazza ,

Guanya Shi ,

Karen Liu

Embodied AI and Robotics Imitation Learning Visual Servoing SRT Task Decomposition RP RKDM

基于感知的人形机器人运动-操作协同控制，需要将第一人称视角的观测数据与任务指令映射为全身运动轨迹。学习这一映射关系，需同时具备同步的第一人称图像、自然语言指令以及适配机器人执行的运动学轨迹；然而，目前尚无任何现有数据源能在大规模尺度上提供这三者完整配套的数据元组。为突破这一瓶颈，我们提出在重建的三维场景中合成生成视觉-语言-运动学（VLK）监督信号。我们的处理流程首先利用三维高斯溅射（3D Gaussian Splatting）技术重建具有真实度量尺度的室内环境；继而借助场景先验信息（privileged scene information），自动合成导航与物体交互的运动轨迹；最后，再回溯式地渲染出与之严格配对的第一人称视角观测图像。整个过程完全无需人工干预，共生成48,000条配对轨迹。基于该数据集，我们训练出一个VLK策略模型，可预测短时域内的全身运动学轨迹；再由一个全身运动追踪器（whole-body tracker）将这些预测轨迹实时转换为物理人形机器人可执行的动作。我们在真实的Unitree G1人形机器人平台上开展了实验验证，任务涵盖自主导航与单物体搬运，结果表明：在重建场景中合成生成的交互数据，能够为“仿真到现实”（sim-to-real）的感知驱动型人形机器人运动-操作协同控制提供高效且可靠的监督信号。项目网站：https://vision-language-kinematics.github.io/

Perception-based humanoid loco-manipulation requires connecting egocentric observations and task instructions to whole-body motion. Learning this mapping requires synchronized egocentric images, language commands, and robot-compatible kinematic trajectories, yet no existing data source provides this complete tuple at scale. We address this bottleneck by generating vision-language-kinematics (VLK) supervision synthetically in reconstructed scenes. Our pipeline leverages 3D Gaussian Splatting to reconstruct metric-scale indoor environments, synthesizes navigation and object-interaction trajectories using privileged scene information, and renders paired egocentric observations after the fact. We produce 48,000 paired trajectories with no human intervention and train a VLK policy that predicts short-horizon whole-body kinematic trajectories. A whole-body tracker converts these predictions into actions on the physical humanoid. We evaluate on the physical Unitree G1 performing navigation and single-object transport, demonstrating that synthesized interactions in reconstructed scenes provide effective supervision for sim-to-real perception-based humanoid loco-manipulation. Project Website: https://vision-language-kinematics.github.io/

许愿