MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation

AK's Picks GenAI Diffusion SCCE CV NeRF 3DGS
2026年06月24日
仅凭单目参考视频,沿目标相机轨迹合成新视角视频,既需保证与参考视频在几何结构上的一致性,又需忠实还原其运动特性。目前基于显式三维表征的方法受限于现成三维重建模块的精度,而这些模块在处理单目视频中的动态物体时,往往生成不准确的几何结构。相比之下,仅依赖相机参数条件化的方法虽能实现较高的视觉质量,却常常难以维持几何结构与运动模式的一致性。本文提出 MVTrack4Gen(面向新视角生成的多视角点跟踪方法),一种具备运动感知能力的训练框架,它将多视角点跟踪作为额外的几何与运动监督信号,引入仅以相机参数为条件的新视角视频扩散模型中。我们的核心发现是:特定注意力层中蕴含着强烈的对应关系线索——查询特征(query features)会关注跨视角、跨时间步中几何位置相对应的关键特征(key features);而此类对应关系一旦错位,便会引发运动不一致问题。基于这一观察,我们将这些特征路由至一个辅助性的多视角跟踪头,并联合优化扩散模型与点跟踪目标函数。通过显式强化这些运动感知型对应关系,MVTrack4Gen 有效提升了现有模型对参考视角中运动模式的跟随能力,并显著增强了跨视角的几何一致性。在多种基准数据集上的实验表明,本方法在几何一致性指标上达到当前最优水平,同时在相机轨迹精度方面也具备强竞争力。
Synthesizing a novel-view video from a monocular reference video along a target camera trajectory requires both geometric consistency and motion fidelity with respect to the reference video. Existing methods based on explicit 3D representations are limited by the accuracy of off-the-shelf reconstruction modules, which often produce inaccurate geometry for dynamic objects in monocular videos. In contrast, camera-conditioning-only methods can achieve high visual quality but often struggle to preserve geometric and motion consistency. In this work, we introduce MVTrack4Gen (Multi-View point Tracking for Novel-View Generation), a motion-aware training framework that leverages multi-view point tracking as an additional geometric and motion supervision signal for camera-conditioning-only novel-view video diffusion models. Our key finding is that specific attention layers encode strong correspondence cues, where query features attend to key features at geometrically corresponding locations across views and over time, and the misalignment of these correspondences causes motion inconsistency. Based on this observation, we route these features into an auxiliary multi-view tracking head and jointly train the diffusion model with a point-tracking objective. By explicitly strengthening these motion-aware correspondences, MVTrack4Gen improves existing models to better follow the motion in the reference view and maintain cross-view geometric consistency. Across diverse benchmarks, our method achieves state-of-the-art geometric consistency and competitive camera accuracy.
许愿