来自今天的爱可可AI前沿推介

[CV] Panoptic Lifting for 3D Scene Understanding with Neural Fields

Y Siddiqui, L Porzi, S R Buló, N Müller, M Nießner, A Dai, P Kontschieder
[Meta Reality Labs Zurich & Technical University of Munich]

基于神经辐射场的3D场景理解全景提升

要点:

  1. 提出Panoptic Lifting，一种新的将2D机器生成全景标签提升到隐式3D体表示的全景辐射场表示方法；
  2. 强大的范式处理机器生成标签中固有的噪声和不一致性；
  3. 在提供在实际场景工作能力的同时，在数据集中取得了显着的改进。

摘要：
提出了全景提升，一种从实际场景图像中学习全景3D体表示的新方法。经过训练，该模型可以从新的视角渲染彩色图像以及3D一致的全景分割。与直接或间接使用3D输入的现有方法不同，该方法只需要从预训练网络推断出机器生成2D全景分割掩码。本文的核心贡献是基于神经场表示的全景提升方案，该方案生成场景的统一和多视图一致的3D全景表示。为了解释跨视图的2D实例标识符的不一致，根据模型的当前预测和机器生成的分割掩码的损失解决了线性分配，使得能以一致的方式将2D实例提升到3D。本文进一步提出，使该方法对含噪的机器生成标签更鲁棒，包括用于置信估计的测试时增强、段一致损失、有界分割场和梯度停止。实验结果验证了对具有挑战性的Hypersim、Replica和ScanNet数据集的方法，场景级PQ比最新水平提高了8.4、13.8和10.6%。

We propose Panoptic Lifting, a novel approach for learning panoptic 3D volumetric representations from images of in-the-wild scenes. Once trained, our model can render color images together with 3D-consistent panoptic segmentation from novel viewpoints. Unlike existing approaches which use 3D input directly or indirectly, our method requires only machine-generated 2D panoptic segmentation masks inferred from a pre-trained network. Our core contribution is a panoptic lifting scheme based on a neural field representation that generates a unified and multi-view consistent, 3D panoptic representation of the scene. To account for inconsistencies of 2D instance identifiers across views, we solve a linear assignment with a cost based on the model's current predictions and the machine-generated segmentation masks, thus enabling us to lift 2D instances to 3D in a consistent way. We further propose and ablate contributions that make our method more robust to noisy, machine-generated labels, including test-time augmentations for confidence estimates, segment consistency loss, bounded segmentation fields, and gradient stopping. Experimental results validate our approach on the challenging Hypersim, Replica, and ScanNet datasets, improving by 8.4, 13.8, and 10.6% in scene-level PQ over state of the art.

论文链接：https://arxiv.org/abs/2212.09802
图片
图片
图片
图片

内容中包含的图片若涉及版权问题，请及时与我们联系删除