Building Rome from a Single Image

AK's Picks CV Monocular or Binocular Depth Estimation and Occlusion Handling VSER
单张图像场景生成的目标是从一张输入图像中重建出完整的三维场景网格,包括相机视野中未观测到的表面区域。尽管预训练的三维物体生成器已编码了较强的形状先验知识,但它们主要面向固定标准体积内的孤立物体,并且大多聚焦于室内场景——这是因为适用于室外场景的多样化三维数据极为匮乏。在本研究中,我们提出一种方法,对以物体为中心的生成器(例如Trellis 2)进行重构,使其既能处理室内场景,也能处理室外场景,同时完整保留其原有的形状先验能力。具体而言,我们通过以下三方面实现这一目标:(a)将整个场景划分为自适应尺度的体素块(chunks),各块尺寸随其到相机的距离动态调整:靠近相机的区域采用更小的块以保留精细几何细节,而远处的结构(如建筑物)则由更大的块覆盖;(b)显式建模图像特征与三维空间之间的对应关系,即通过对图像特征进行深度提升(lifting),并使模型明确感知自由空间、已观测表面以及未观测区域;(c)合成了约4000个室外场景用于扩充训练数据,以弥补现有场景数据集普遍局限于室内的不足。在Tanks and Temples、ScanNet++以及野外真实图像上的实验结果表明,我们的方法在几何精度与感知质量两方面均显著优于所有基线方法,且在室内与室外场景上均保持稳定优越性。
Single-image scene generation aims to produce a complete 3D scene mesh from a single image, including surfaces the camera did not observe. While pretrained 3D object generators encode a strong shape prior, they are mainly designed for isolated objects in a fixed canonical volume and focus mostly on indoor scenes, since diverse 3D data for outdoor scenes are quite limited. In this work, we present a method that redesigns such an object-centric generator, e.g., Trellis 2, to work on both indoor and outdoor scenes while retaining its prior. We accomplish this by (a) partitioning the scene into adaptive chunks that scale relative to the distance to the camera; nearby chunks have a smaller size to keep the finer detail, while distant structures, e.g., buildings, are covered by large chunks; (b) making the generator capture explicit 2D-3D correspondence by lifting image features and making the model aware of the free space, observed surface, and unobserved region; (c) synthesizing around 4,000 outdoor scenes to broaden the training data, as existing scene datasets are largely indoor. Experiments on Tanks and Temples, ScanNet++, and in-the-wild images show that our method outperforms all baselines in geometric accuracy and perceptual quality across both indoor and outdoor scenes.
许愿