What 30,000 Hours of Ego-centric Video Does Not Teach

Embodied AI and Robotics World Models RKDM ML SSLRO Generative Models CV ARSDM
世界模型为基于物理的仿真器提供了一种颇具前景的替代方案,但距离实际部署仍有相当大的差距。我们探究了以自我为中心的人类视频数据规模的扩大能在多大程度上提升此类模型的性能,所用数据集涵盖逾1,000种场景类型、来自14,000名贡献者的总计30,000小时视频。我们并未依赖难以解读的下游任务指标,而是直接在一个极具挑战性的分布外(out-of-distribution)基准测试上,评估智能体(agent)建模与物体交互建模的保真度。将训练数据量扩大100倍后,二者性能均有所提升,但提升幅度极不均衡:智能体建模效果良好,而物体建模的保真度仍显著偏低,且提升速度缓慢。我们进一步证明,智能体建模能力的提升未必依赖于数据量的增长;通过精心设计视觉条件输入机制,仅需少量数据即可使智能体建模保真度趋于饱和。这使得我们能够单独衡量物体建模保真度,并由此发现其自身的饱和点。随后,我们提出一种新型监督机制,将模型容量从场景外观建模转向物体动力学建模,从而在一定程度上提升了物体建模保真度——尽管与理想水平之间仍存在明显差距。最后,我们的结论同样适用于下游的人形机器人(humanoid)建模任务。总体而言,本研究结果表明:扩大以自我为中心的视频数据规模,虽已使智能体建模性能逼近其理论极限,却远未同步提升模型对“外部世界”(尤其是物体动态行为)的刻画能力;而要弥合这一差距,关键不仅在于数据量的多少,更取决于模型的训练方式。
World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment. We ask how far scaling ego-centric human video takes them, using a dataset of 30,000 hours spanning over 1,000 scene types and 14,000 contributors. Rather than relying on opaque downstream metrics, we directly evaluate agent and object-interaction fidelity on a challenging out-of-distribution benchmark. Increasing training data by 100x improves both, but unevenly: the agent is modeled well, while object fidelity remains far lower and improves slowly. We show that the agent gains need not come from data, and a careful visual conditioning design saturates fidelity with a fraction of it, which lets us measure object fidelity on its own and discover its saturation point. We then introduce a supervision scheme that shifts capacity from scene appearance toward object dynamics, improving object fidelity though a substantial gap remains. Finally, our conclusions transfer to downstream humanoid modeling. Overall, our results suggest that scaling ego-centric data brings agent modeling close to its limit while leaving its effects on the world far behind, and that closing this gap will depend on how models are trained, not only on how much data they see.
许愿