MIME：人体感知3D场景生成

来自今天的爱可可AI前沿推介

[CV] MIME: Human-Aware 3D Scene Generation

H Yi, C P. Huang, S Tripathi, L Hering, J Thies, M J. Black
[Max Planck Institute for Intelligent Systems & Adobe Inc]

MIME：人体感知3D场景生成

简介：人与环境交互，其中就包含了有关布局和对象放置的信息。MIME(挖掘交互和运动以推断3D环境)是一种生成模型，可在给定3D人体运动的情况下生成3D室内场景和家居布局。本文构建了一个具有交互和自由空间人体的数据集，可以从运动捕捉数据生成逼真的场景。

摘要：生成由移动的人体占据的逼真3D世界在游戏、建筑和合成数据创建中有许多应用。但是生成这样的场景是昂贵且劳动密集的。最近的工作在给定3D场景的情况下生成人体姿态和动作。本文则在给定3D人体运动的情况下生成3D室内场景。此类运动可以来自存档运动捕捉或来自佩戴在身体上的IMU传感器，从而有效地将人体运动转变为3D世界的“扫描仪”。直观上，人体运动表示房间内的自由空间，人体接触表示支持坐、卧或触摸等存在的表面或物体。本文提出 MIME(挖掘交互和运动以推断3D环境)，这是一种室内场景的生成模型，可生成与人体运动一致的家具布局。MIME使用自回归transformer架构，将场景中已经生成的对象以及人体运动作为输入，并输出下一个可靠的对象。为了训练 MIME，本文通过使用3D人体填充3D FRONT场景数据集来构建数据集。实验表明，与不了解人体运动的最新场景生成方法相比，MIME生成的3D场景更加多样化且合理。

Generating realistic 3D worlds occupied by moving humans has many applications in games, architecture, and synthetic data creation. But generating such scenes is expensive and labor intensive. Recent work generates human poses and motions given a 3D scene. Here, we take the opposite approach and generate 3D indoor scenes given 3D human motion. Such motions can come from archival motion capture or from IMU sensors worn on the body, effectively turning human movement in a "scanner" of the 3D world. Intuitively, human movement indicates the free-space in a room and human contact indicates surfaces or objects that support activities such as sitting, lying or touching. We propose MIME (Mining Interaction and Movement to infer 3D Environments), which is a generative model of indoor scenes that produces furniture layouts that are consistent with the human movement. MIME uses an auto-regressive transformer architecture that takes the already generated objects in the scene as well as the human motion as input, and outputs the next plausible object. To train MIME, we build a dataset by populating the 3D FRONT scene dataset with 3D humans. Our experiments show that MIME produces more diverse and plausible 3D scenes than a recent generative scene method that does not know about human movement. Code and data will be available for research at this https URL.

论文链接：https://arxiv.org/abs/2212.04360

内容中包含的图片若涉及版权问题，请及时与我们联系删除