CLiC: Concept Learning in Context

Multimodal Intelligence FGCMA TGIE GenAI TDIG
本文解决了从一张图片中学习一个物体的局部视觉模式,并生成描绘具有该模式物体的图像的挑战。学习一个局部概念并将其放置在目标图像中的物体上是一项非常困难的任务,因为物体可能具有不同的方向和形状。我们的方法建立在视觉概念学习的最新进展之上。它涉及从源图像获取视觉概念(例如装饰)并随后将其应用于目标图像中的物体(例如椅子)。我们的关键思想是进行上下文概念学习,在它们所属的物体的更广泛上下文中获取局部视觉概念。为了定位概念学习,我们使用包含掩模内部概念和周围图像区域的软掩模。我们通过图像内的对象生成展示了我们的方法,展示了在上下文学习的概念中合理嵌入的可能性。我们还介绍了将获取的概念定向到目标图像中特定位置的方法,采用交叉注意机制,并建立源对象和目标对象之间的对应关系。我们通过定量和定性实验以及与基线技术的比较证明了我们方法的有效性。
This paper addresses the challenge of learning a local visual pattern of an object from one image, and generating images depicting objects with that pattern. Learning a localized concept and placing it on an object in a target image is a nontrivial task, as the objects may have different orientations and shapes. Our approach builds upon recent advancements in visual concept learning. It involves acquiring a visual concept (e.g., an ornament) from a source image and subsequently applying it to an object (e.g., a chair) in a target image. Our key idea is to perform in-context concept learning, acquiring the local visual concept within the broader context of the objects they belong to. To localize the concept learning, we employ soft masks that contain both the concept within the mask and the surrounding image area. We demonstrate our approach through object generation within an image, showcasing plausible embedding of in-context learned concepts. We also introduce methods for directing acquired concepts to specific locations within target images, employing cross-attention mechanisms, and establishing correspondences between source and target objects. The effectiveness of our method is demonstrated through quantitative and qualitative experiments, along with comparisons against baseline techniques.
许愿