Geometric Reciprocity: Unlocking Self-Supervision for Stereoscopic Video Generation

GenAI SCCE Inpainting ML SSL CV Monocular or Binocular Depth Estimation and Occlusion Handling
2026年07月06日
单目转双目技术通过从二维视频中合成立体影像,从而实现沉浸式三维视觉体验。在当前主流的基于深度图像的渲染(DIBR)方法中,遮挡区域(disocclusion)的立体补全(stereo inpainting)已成为制约整体性能的关键瓶颈。基于训练的方法虽能获得更优的合成质量,却严重依赖于稀缺的真实双目视频对,或存在明显域偏移(domain gap)的合成数据。为此,我们首次提出一种完全自监督的学习框架,仅利用单目视频,并借助循环一致性(cycle consistency)进行训练。本工作的核心贡献是提出了“几何互易性定理”(Geometric Reciprocity Theorem, GRT):在最近邻DIBR建模下,合成目标视图时产生的遮挡区域掩模,恰好等于将目标视图反向扭曲(warping back)至源视图过程中所丢失像素的掩模;由此,可在测试阶段直接、解析地从单目图像中计算出遮挡区域掩模。该性质确保了所采用的扭曲模型在训练与测试阶段具有一致性,从而支持利用海量无标注单目视频开展自监督学习,并在性能上显著超越现有无需训练(training-free)及有监督的最先进方法。项目主页:https://visual-ai.github.io/grt/
Monocular-to-stereo conversion synthesizes stereoscopic content from 2D videos for immersive 3D experiences. In modern Depth-Image-Based Rendering (DIBR) approaches, stereo inpainting of disocclusions is the critical bottleneck. Training-based methods achieve superior quality but rely on scarce stereo pairs or synthetic data with domain gaps. We address this through the first self-supervised framework learning from monocular videos via cycle consistency. Our key contribution is the Geometric Reciprocity Theorem (GRT): under the nearest-neighbor DIBR formulation, the disocclusion mask when synthesizing a target view equals the mask of pixels lost when warping back from target to source, enabling analytical computation of test-time disocclusion masks directly from monocular images. This yields train-test consistency for the stated warping formulation, supporting self-supervised learning from unlimited monocular videos and substantial improvements over training-free and supervised state-of-the-art methods. Project page: https://visual-ai.github.io/grt/
许愿