ARM: An AutoRegressive Large Multimodal Model with Unified Discrete Representations

AK's Picks Multimodal Intelligence Vision-Language Pre-training MHD TGIE GenAI TDIG ML RL
2026年06月09日
本文提出了ARM(一种基于离散表征的自回归模型),该模型在“下一个词元预测”(next-token prediction)框架下,统一实现了图像理解、生成与编辑三大任务。ARM的构建包含三项核心工作:首先,我们训练了一种离散化的语义视觉分词器(semantic visual tokenizer),可将图像映射为紧凑的词元序列;该分词器采用多目标监督方式进行训练,协同优化语义判别能力、语言对齐性以及重建保真度,从而在共享隐空间中支撑多样化任务。依托这一分词器,我们在大规模图文混合词元序列上训练了一个参数量达70亿(7B)的自回归大模型,使其自然习得跨模态的视觉-语言感知与生成能力。最后,为显著提升文本到图像生成及指令引导式图像编辑任务中与人类偏好一致的行为表现,ARM进一步引入强化学习(RL),直接针对任务级目标(如图像视觉质量、指令遵循度、编辑一致性)进行端到端优化。出人意料的是,实验结果表明:强化学习不仅大幅提升了目标任务性能(例如,WISE整体得分从0.50提升至0.56,GEdit-Bench-EN的G_O指标从5.75提升至6.68),还意外催生了文本到图像生成与图像编辑两大任务之间的跨任务协同效应。综上所述,这些发现共同印证:当自回归建模范式与高质量表征能力及偏好优化机制相结合时,便能构成一种可扩展的多模态智能基础架构。代码开源地址:https://github.com/wdrink/ARM。
This paper introduces ARM, a discrete representation-based AutoRegressive Model that unifies image understanding, generation, and editing within a next-token prediction framework. ARM is built on three efforts: first, we train a discrete semantic visual tokenizer that maps images into compact token sequences. Our tokenizer is supervised with multiple objectives that jointly promote semantic discriminability, language alignment and faithful reconstruction, thereby supporting diverse tasks in a shared latent space. With this, we train a 7B autoregressive model over large-scale text and image token sequences, seamlessly developing vision-language perception and generation capabilities. Finally, to further improve preference-aligned behavior for text-to-image generation and instruction-guided editing, ARM applies reinforcement learning (RL) to optimize task-level objectives such as visual quality, instruction adherence, and edit consistency. Surprisingly, the results show that RL not only substantially improves performance on the target tasks (e.g., raising WISE overall from 0.50 to 0.56, GEdit-Bench-EN G_O from 5.75 to 6.68), but also induces cross-task synergy between text-to-image generation and editing. Collectively, these findings highlight autoregressive modeling, when paired with strong representations and preference optimization, as a scalable foundation for multimodal intelligence. Code: https://github.com/wdrink/ARM.
许愿