Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis

Keerthi Kaashyap ,

Dennis Anthony ,

Akshay Krishnan ,

Nhi Ngoc Nguyen ,

Jeremy Collins ,

James Hays ,

Shreyas Kousik ,

Animesh Garg

ML TAVR SSLRO CV NeRF VSER

本文探讨了新视角合成（Novel View Synthesis, NVS）在几何表征学习中的作用。理论上，NVS 应能对三维场景结构进行推理，从而习得可迁移的多视角几何表征。然而，当前基于编码器的 NVS 方法所生成的表征质量较差。这并非源于监督信号的缺失，而主要归因于一些看似细微却影响深远的网络架构设计：一是解码器具有过强的空间表达能力（spatially expressive decoders），削弱了场景编码器本身的表征能力；二是重建目标设定在低层次的像素空间（low-level pixel-space targets），阻碍了高层语义特征的学习。为此，我们提出了 SNAP——一种自监督的编码器–解码器式 Transformer 模型，通过引入姿态条件化的局部解码器（pose-conditioned local decoder）与潜空间重建目标（latent-space reconstruction objective），从根源上解决了上述两个问题。SNAP 具有任务无关性（task-agnostic），实验表明，其性能可媲美专为几何任务设计、依赖显式几何监督的模型。此外，在视觉定位、位姿估计、点对应匹配、深度估计和机器人操作共五项下游任务中，SNAP 的表现亦优于现有各类自监督表征方法。尤为值得注意的是，SNAP 所学习的图像块（patch）特征自发涌现出显著的视角不变性（emergent viewpoint invariance），其水平已接近高度依赖大量标注数据与算力的强监督模型，而其训练所需的计算资源与数据规模却远低于后者。当相机发生较大位移、传统二维表征性能急剧下降甚至完全失效时，SNAP 的性能退化则更为平缓——这一现象揭示出：主动限制解码器的表达能力，反而能有效避免可迁移几何结构被过度抑制。https://snap-nvs.github.io

This paper examines the role of Novel View Synthesis (NVS) in geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view geometric representations. Yet, existing encoder-based NVS methods yield poor representations. This is not because of a lack of supervisory signal, but rather due to inconspicuous architectural choices: \textit{spatially expressive decoders} that dilute representational capabilities of the scene encoder, and \textit{low-level pixel-space targets} that hinder feature learning. We present SNAP, a self-supervised encoder-decoder transformer that addresses both through a pose-conditioned local decoder and a latent-space reconstruction objective. SNAP is task agnostic, and we show that it is competitive with special-purpose geometry-supervised methods. SNAP also performs competitively against self-supervised representations across five tasks: visual localization, pose estimation, point correspondence, depth estimation, and robot manipulation. Remarkably, SNAP's patch features exhibit emergent viewpoint invariance that approaches heavily supervised models despite lower compute and data budgets. Under camera shifts where standard 2D representations collapse, SNAP degrades more gracefully, revealing that restricting decoder expressivity actively prevents the suppression of transferable geometric structure. https://snap-nvs.github.io

许愿