OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes

AK's Picks GenAI Diffusion TDIG CV NeRF 3DGS
2025年10月30日
目前构建3D场景主要有两种流行方法:程序化生成和基于2D的提升(2D lifting)。其中,基于全景图的2D提升技术近年来展现出巨大潜力,它利用强大的2D生成先验知识,能够生成沉浸感强、逼真且多样化的3D环境。在本研究中,我们进一步推进了这一技术,旨在生成适用于基于物理的渲染(PBR)、重光照和仿真的、可直接用于图形处理的3D场景。我们的核心思路是将现有的2D生成模型重新用于全景图中几何结构、纹理以及PBR材质的感知任务。与现有2D提升方法主要关注外观生成而忽略对内在属性感知不同,我们提出了OmniX——一个通用且统一的框架。该框架基于轻量高效且跨模态的适配器结构,复用2D生成先验知识,以支持广泛的全景视觉任务,包括全景感知、生成与补全。此外,我们构建了一个大规模的合成全景数据集,其中包含来自多样化室内外场景的高质量多模态全景图像。大量实验表明,我们的模型在全景视觉感知和可图形化3D场景生成方面均表现出色,为创建沉浸式且符合物理规律的虚拟世界开辟了新的可能。
There are two prevalent ways to constructing 3D scenes: procedural generation and 2D lifting. Among them, panorama-based 2D lifting has emerged as a promising technique, leveraging powerful 2D generative priors to produce immersive, realistic, and diverse 3D environments. In this work, we advance this technique to generate graphics-ready 3D scenes suitable for physically based rendering (PBR), relighting, and simulation. Our key insight is to repurpose 2D generative models for panoramic perception of geometry, textures, and PBR materials. Unlike existing 2D lifting approaches that emphasize appearance generation and ignore the perception of intrinsic properties, we present OmniX, a versatile and unified framework. Based on a lightweight and efficient cross-modal adapter structure, OmniX reuses 2D generative priors for a broad range of panoramic vision tasks, including panoramic perception, generation, and completion. Furthermore, we construct a large-scale synthetic panorama dataset containing high-quality multimodal panoramas from diverse indoor and outdoor scenes. Extensive experiments demonstrate the effectiveness of our model in panoramic visual perception and graphics-ready 3D scene generation, opening new possibilities for immersive and physically realistic virtual world generation.
许愿