MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

Embodied AI and Robotics World Models Trajectory Prediction CV ARSDM Other CV
2026年07月17日
人类仅通过被动观察，就能推断物体可能如何运动：例如杯子可能被拿起、抽屉可能滑动打开、盖子可能旋转闭合。这类预测揭示了与物理世界交互时所必需的运动后果，是人类在真实环境中采取行动的基础。我们研究如何仅从普通单目视频（即普通人拍摄的人-物交互视频）中学习这种运动预测能力。给定一段较短的已观测视频片段，MotionForesight 能够预测被操作物体表面各点在未来一段时间内的三维运动轨迹。该方法将人-物交互预测建模为以物体为中心的三维运动预测任务，且无需对物体本身的属性（如形状、材质、类别等）作任何先验假设。我们的核心洞见在于：现有的视频预测模型已隐式编码了大量关于物体在人类交互过程中如何运动的丰富先验知识。我们所做的工作，是将这些先验知识从原本用于像素级视频重建的任务，重新导向未来三维场景光流（scene flow）的预测。具体而言，我们首先基于一个预训练的视频基础模型构建了一个稠密三维追踪器；随后，利用完整视频片段生成伪真实轨迹（pseudo-ground-truth tracks）作为监督信号；最终，仅使用观测到的视频帧来训练运动预测器。在训练过程中，我们用可学习的掩码隐变量（learned mask latents）替代未来帧的 RGB 图像和几何信息，并额外引入一个轻量级适配器（lightweight adapter），将原本面向回溯式追踪（retrospective tracking）的表征转化为前向运动预测表征——与此同时，保持庞大的视频编码器和追踪模块完全冻结不变。仅需 4 万段人类交互视频、且不依赖语言描述等任何辅助输入，MotionForesight 即可在大量分布外（out-of-distribution）场景中实现良好泛化，涵盖种类繁多的未知物体、多样化环境、任意视角以及各类交互行为。其性能甚至显著优于参数量更大、且使用超百万段训练视频的对比模型。上述结果表明，我们能够高效地将视频模型中蕴含的先验知识，迁移并重构为面向具身智能（embodied intelligence）所需的显式几何运动预测能力。https://motionforesight.github.io/
Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/
许愿