Reading, Writing, Sharing, and Interacting with the World through Video

Embodied AI and Robotics World Models Multimodal Intelligence LVTU GenAI TDIG SCCE
2026年10月22日
视频基础模型在视觉序列的理解与生成方面已取得迅猛进展,而视频世界模型则致力于捕捉其所表征世界的内在动态规律。尽管二者目标不同,但均聚焦于视频观测与持续演化的世界之间的关系。本综述以“世界建模”为统一视角,系统梳理和审视这一领域的发展脉络。我们将所观测的视频视为潜在时序世界的一种部分性、有损性且按时间顺序排列的投影,并据此将相关文献围绕四项互补性操作进行组织:**读取(reading)**——从观测视频中推断世界属性;**书写(writing)**——依据指定或习得的世界条件合成视频;**共享(sharing)**——通过共用的建模范式,实现对世界理解与视频生成能力的有机融合;**交互(interacting)**——在施加动作及持续接收新观测的前提下,预测并更新世界状态。这一视角明确区分了观测或生成序列所体现的**时间连贯性(temporal coherence)**与潜在世界状态所要求的**一致性(consistency)**。我们进一步从任务类型、表征形式、模型架构、数据集及评测基准等多个维度出发,系统梳理研究重心的演进趋势:即由早期关注片段级识别(clip-level recognition)与感知保真度(perceptual fidelity),逐步转向强调持久性世界状态(persistent state)、可控的动态演化(controllable evolution)、符合物理规律与因果逻辑的有效性(physical and causal validity),以及以交互过程和下游决策为根基的评估范式(evaluation grounded in interaction and downstream decisions)。最后,我们凝练出该领域当前亟待突破的关键挑战。
Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.
许愿