PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation

AK's Picks GenAI Diffusion CV Monocular or Binocular Depth Estimation and Occlusion Handling
当前最先进的单图像三维重建方法,往往依赖于复杂的混合网络架构与损失函数设计,或需将几何信息压缩至潜在空间,以便利用预训练的潜在扩散模型。在本研究中,我们证明了此类繁复的架构设计与精巧的损失函数构建实属不必。我们提出了一种极简主义的像素空间扩散Transformer模型,其主干基于标准的ViT(视觉Transformer)架构,直接作用于原始三维点云图(point map)的图像块,并以预训练的DINOv3模型所提取的图像token作为条件输入。与现有潜在扩散方法不同,我们的扩散主干网络完全从零开始端到端训练,无需借助点云图tokenizer对几何信息进行离散化编码。尽管结构极为简洁,本方法在性能上仍超越了各类复杂的基于潜在空间的扩散模型,同时显著优于各类混合式架构方案。尤为值得注意的是,该方法生成的几何结构更为锐利清晰,在高度模糊或歧义性强的区域(例如透明物体)中亦展现出更强的鲁棒性。
State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures and loss functions, or compress geometry into latent spaces in order to leverage pre-trained latent diffusion models. In this work, we show that such architectural overhead and intricate loss formulations are unnecessary. We introduce a minimalist pixel-space Diffusion Transformer, built on a plain ViT, that operates directly on raw 3D point map patches and is conditioned on image tokens from a pre-trained DINOv3. Unlike existing latent diffusion approaches, we train our diffusion backbone entirely from scratch, eliminating the need for point map tokenizers. Despite its simplicity, our approach surpasses complex latent-based diffusion models while remaining significantly simpler than hybrid alternatives. Notably, it produces sharper geometric structure and is more robust in highly ambiguous regions, such as transparent objects.
许愿