VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization

我们提出了VQ-NeRF，这是一个双分支神经网络模型，它结合了向量量化（VQ）来分解和编辑三维场景中的反射场。传统的神经反射场仅使用连续表示来建模三维场景，尽管现实中的物体通常由离散材料组成。这种缺乏离散化可能导致材料分解有噪声和材料编辑复杂。为了解决这些限制，我们的模型由一个连续分支和一个离散分支组成。连续分支遵循传统流程以预测分解的材料，而离散分支使用VQ机制将连续材料量化为单个材料。通过离散化材料，我们的模型可以减少分解过程中的噪声，并生成离散材料的分割图。可以通过单击分割结果的相应区域轻松选择特定材料进行进一步编辑。此外，我们提出了一种基于丢失的VQ码字排名策略来预测场景中的材料数量，从而减少材料分割过程中的冗余。为了提高可用性，我们还开发了一个交互式界面来进一步辅助材料编辑。我们在计算机生成的和真实世界的场景上评估了我们的模型，展示了其优越的性能。据我们所知，我们的模型是第一个能够在三维场景中实现离散材料编辑的模型。

The paper proposes a model to address the limitations of conventional neural reflectance fields that use only continuous representations to model 3D scenes, resulting in noisy material decomposition and complicated material editing. The model aims to enable discrete material editing in 3D scenes.

The model consists of a continuous branch and a discrete branch that uses Vector Quantization (VQ) to quantize continuous materials into individual ones. By discretizing the materials, the model can reduce noise in the decomposition process and generate a segmentation map of discrete materials. The paper also proposes a dropout-based VQ codeword ranking strategy to predict the number of materials in a scene, which reduces redundancy in the material segmentation process.

The paper evaluates the proposed model on both computer-generated and real-world scenes, demonstrating its superior performance. The model also includes an interactive interface to assist material editing. The paper provides an in-depth analysis of the model's performance and limitations and suggests potential future research directions. The code and data are available online.

Related work in this field includes 'NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis' by Mildenhall et al., 'GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis' by Schwartz et al., and 'PixelNeRF: Neural Radiance Fields from One or Few Images Using Pixel-Wise Experts' by Sitzmann et al.