Thinking in Boxes: 3D Editing in Real Images Made Easy

GenAI SCCE Inpainting CV NeRF 3DGS
2026年06月18日
文本及二维条件控制接口在图像编辑中对空间变换的调控能力较弱,且存在歧义性——尤其在物体发生大幅运动或相机视角发生变化时,这一问题更为突出。以往研究虽曾采用三维基元(如三维边界框)作为条件输入,但仅将其用作粗略的定位提示信号,用以大致指示物体所在位置,而并未将其用于明确指定具体的几何变换。本文则将三维边界框作为结构化编辑规范:用户只需提供编辑操作前后的输入与输出三维边界框,即可将图像编辑任务建模为一个定义明确的几何求解问题。我们提出的这种“以框思考”(thinking in boxes)交互方式,通过对每个边界框面进行颜色编码以直观表达其三维朝向,从而实现对真实图像中平移、旋转、缩放以及视点变化的精确控制;同时,在保持场景整体结构与物体身份一致性的前提下,还能重建出原先图像中未曾可见的物体区域。为使所生成的几何变换与真实场景外观紧密耦合,我们引入了一个深度对齐的平面地面作为全局参考坐标系,并辅以基于深度信息的阴影提示进行视觉增强。在此结构化约束条件下,图像生成器能够针对大幅几何变换生成高度一致的结果。本系统采用两阶段训练策略:首先在合成的多物体三维场景上进行预训练,随后在Objectron数据集提供的少量真实世界视频片段上进行微调,从而使其具备良好的泛化能力,可稳健处理复杂、开放环境下的真实图像。本方法直接作用于原始真实照片,在大幅三维编辑任务上显著优于近期各类前沿方法。
Text and 2D-conditioning interfaces provide weak, ambiguous control over spatial transformations in image editing -- particularly under large object motions and camera changes. Prior work has used 3D primitives such as boxes, but only as loose conditioning signals indicating approximate object location rather than specifying the transformation. We instead use 3D boxes as structured specifications: the user provides the input and output boxes of the edit, casting editing as a well-posed geometry problem. This ``thinking in boxes'' interface, where each box face is color-coded to convey 3D orientation, gives precise control over translation, rotation, scaling, and viewpoint changes in real images while preserving scene and object identity, and recovering previously unseen object regions. To ground transformations in scene appearance, we introduce a depth-aligned planar floor as a global reference frame, shaded with depth-aware cues. Conditioned on this structure, an image generator produces consistent results under large transformations. Trained in two stages -- on synthetic multi-object scenes and a small set of real-world videos from Objectron -- the system generalizes to complex, in-the-wild real images. Our method operates directly on real photographs and substantially outperforms recent state-of-the-art methods on large 3D edits.
许愿