来自今天的爱可可AI前沿推介

[CV] Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program

T Luo, H Lee, J Johnson
[University of Michigan]

神经形状编译器：文本、点云和程序间转换的统一框架

要点:

  1. 提出神经形状编译器来转换三种形状抽象：文本，点云和程序；
  2. 在PointVQVAE的帮助下，通过统一和可扩展的框架，在文本 =⇒ 点云，点云 =⇒ 文本，点云 =⇒ 程序 和点云补全任务中取得了出色的表现；
  3. 实验表明，神经形状编译器可以从所有异构数据和任务的联合训练中受益。

摘要：
3D形状具有互补的抽象，从底层几何到基于部件层次结构再到语言，这些抽象传达了不同层次的信息。本文提出一种在一对形状抽象之间进行翻译的统一框架：文本⟺ 点云⟺ 程序。提出神经形状编译器，将抽象变换建模为条件生成过程，将三种抽象类型的3D形状转换为统一的离散形状编码，通所提出的形状编码Transformer将每个形状编码转换为其他抽象类型的代码，并对其进行解码以输出目标形状抽象。点云代码由所提出的PointVQVAE以与类无关的方式获得。在Text2Shape、ShapeGlot、ABO、Genre和Program Synthetic数据集上，神经形状编译器显示了文本 =⇒ 点云，点云 =⇒ 文本，点云 =⇒ 程序 和点云补全任务的优势。此外，神经形状编译器受益于所有异构数据和任务的联合训练。

3D shapes have complementary abstractions from low-level geometry to part-based hierarchies to languages, which convey different levels of information. This paper presents a unified framework to translate between pairs of shape abstractions: Text ⟺ Point Cloud ⟺ Program. We propose Neural Shape Compiler to model the abstraction transformation as a conditional generation process. It converts 3D shapes of three abstract types into unified discrete shape code, transforms each shape code into code of other abstract types through the proposed ShapeCode Transformer, and decodes them to output the target shape abstraction. Point Cloud code is obtained in a class-agnostic way by the proposed PointVQVAE. On Text2Shape, ShapeGlot, ABO, Genre, and Program Synthetic datasets, Neural Shape Compiler shows strengths in Text ⟹ Point Cloud, Point Cloud ⟹ Text, Point Cloud ⟹ Program, and Point Cloud Completion tasks. Additionally, Neural Shape Compiler benefits from jointly training on all heterogeneous data and tasks.

论文链接：https://arxiv.org/abs/2212.12952
图片
图片
图片
图片

内容中包含的图片若涉及版权问题，请及时与我们联系删除