3D-aware Conditional Image Synthesis

K Deng, G Yang, D Ramanan, J Zhu
[CMU]

3D感知条件图像合成

要点:

  1. Pix2pix3D 是一种 3D 感知条件生成模型，用于可控的逼真图像合成；
  2. 学习从不同视角合成相应的图像，给定一个 2D 标记图，如分割图或边缘图；
  3. 用神经辐射场为每个 3D 点分配一个标记，除了颜色和密度外，这使得它能同时渲染图像和像素对齐标记图；
  4. 学到的 3D 标记进一步实现了交互式 3D 交叉视图编辑。

一句话总结:
Pix2pix3D 是一种 3D 感知条件生成模型，允许用户在给定的 2D 标记图上渲染不同视角的图像，并能为每个 3D 点分配标记、颜色和密度，实现交互式 3D 跨视图编辑。

论文地址：https://arxiv.org/abs/2302.08509 

We propose pix2pix3D, a 3D-aware conditional generative model for controllable photorealistic image synthesis. Given a 2D label map, such as a segmentation or edge map, our model learns to synthesize a corresponding image from different viewpoints. To enable explicit 3D user control, we extend conditional generative models with neural radiance fields. Given widely-available monocular images and label map pairs, our model learns to assign a label to every 3D point in addition to color and density, which enables it to render the image and pixel-aligned label map simultaneously. Finally, we build an interactive system that allows users to edit the label map from any viewpoint and generate outputs accordingly.


图片
图片
图片
图片

内容中包含的图片若涉及版权问题，请及时与我们联系删除