Masked Visual Actions for Unified World Modeling

AK's Picks Embodied AI and Robotics World Models RC
视频模型能够学习视觉世界中物体运动、交互以及对外力接触响应的丰富先验知识,因而成为构建机器人世界模型极具潜力的基础架构。核心挑战在于:如何以一种既与模型所习得的交互先验所处的视觉空间相一致、又切实扎根于物理操作的方式,向此类模型传达动作指令。为此,我们提出“掩码化视觉动作”(Masked Visual Actions)——一种基于像素空间的动作控制接口,其将动作表征为视频中任意实体的一条部分可见的轨迹。当显式揭示机器人自身的运动轨迹时,该模型便作为前向动力学模型发挥作用,预测场景对底层机器人动作的响应;而当显式揭示目标物体的期望运动轨迹时,同一模型则反向推断出与该结果相一致的机器人行为。仅需利用来自真实视频与仿真的总计15小时掩码化样本进行微调,单个模型检查点即可在多种场景及不同机器人本体上,同时实现优异的视觉保真度与动作可控性。在下游操作任务中,该模型生成的想象 rollout(即预测的未来状态序列)其结果与真实世界执行效果高度相关,可用于策略评估;通过在基于模型的规划中对候选未来状态进行排序,可提升决策质量;此外,该模型还支持逆向建模——即根据期望的物体运动直接合成相应的机器人运动轨迹。
Video models absorb rich priors over how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge is how to communicate action to such models in a form aligned with the visual space in which they learned these interaction priors, yet still grounded in physical manipulation. We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video. Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome. Finetuned with only 15 hours of masked examples from real videos and simulation, a single checkpoint achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments. In downstream manipulation settings, the model produces imagined rollouts whose outcomes correlate with real-world execution for policy evaluation, improves decision making by ranking candidate futures in model-based planning, and supports inverse modeling by synthesizing robot motion from desired object motion.
许愿