OmniGen2: Exploration to Advanced Multimodal Generation

AK's Picks Multimodal Intelligence TGIE CMAG GenAI TDIG Inpainting AI Systems and Hardware MCPKD
在这项工作中,我们推出了OmniGen2,这是一种多功能且开源的生成模型,旨在为各种生成任务提供统一的解决方案,包括文本到图像生成、图像编辑以及上下文内生成。与OmniGen v1不同,OmniGen2为文本和图像模态分别设计了两个独立的解码路径,采用不共享的参数和解耦的图像编码器。这一设计使得OmniGen2可以在现有多模态理解模型的基础上进行构建,而无需重新适配VAE输入,从而保留原有的文本生成能力。为了支持OmniGen2的训练,我们开发了全面的数据构建流程,涵盖图像编辑和上下文内生成所需的数据。此外,我们还为图像生成任务引入了一种专门设计的反思机制,并基于OmniGen2构建了一个专用的反思数据集。尽管参数规模相对适中,OmniGen2在多个任务基准测试中仍取得了具有竞争力的结果,包括文本到图像生成和图像编辑。为了进一步评估上下文内生成(也称为主题驱动任务),我们引入了一个新的基准测试集OmniContext。在开源模型中,OmniGen2在一致性方面达到了最先进的性能水平。我们将发布模型、训练代码、数据集以及数据构建流程,以支持该领域的未来研究。项目页面:https://vectorspacelab.github.io/OmniGen2;GitHub链接:https://github.com/VectorSpaceLab/OmniGen2
In this work, we introduce OmniGen2, a versatile and open-source generative model designed to provide a unified solution for diverse generation tasks, including text-to-image, image editing, and in-context generation. Unlike OmniGen v1, OmniGen2 features two distinct decoding pathways for text and image modalities, utilizing unshared parameters and a decoupled image tokenizer. This design enables OmniGen2 to build upon existing multimodal understanding models without the need to re-adapt VAE inputs, thereby preserving the original text generation capabilities. To facilitate the training of OmniGen2, we developed comprehensive data construction pipelines, encompassing image editing and in-context generation data. Additionally, we introduce a reflection mechanism tailored for image generation tasks and curate a dedicated reflection dataset based on OmniGen2. Despite its relatively modest parameter size, OmniGen2 achieves competitive results on multiple task benchmarks, including text-to-image and image editing. To further evaluate in-context generation, also referred to as subject-driven tasks, we introduce a new benchmark named OmniContext. OmniGen2 achieves state-of-the-art performance among open-source models in terms of consistency. We will release our models, training code, datasets, and data construction pipeline to support future research in this field. Project Page: https://vectorspacelab.github.io/OmniGen2; GitHub Link: https://github.com/VectorSpaceLab/OmniGen2
许愿