Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification

AK's Picks Multimodal Intelligence Vision-Language Pre-training TGIE GenAI Diffusion
2026年06月16日
统一多模态建模旨在将视觉理解与视觉生成整合于同一系统之中。然而,现有方法通常依赖两种彼此独立的视觉分词器(visual tokenizer),导致表征空间被割裂,从而阻碍了真正意义上的统一建模。为此,我们提出 UniAR——一种统一的自回归框架:该框架仅采用单一离散化视觉分词器,作为连接视觉理解与视觉生成的关键桥梁,从而构建共享上下文;在此上下文中,模型可直接解析自身生成的视觉词元(visual token),无需额外的重编码过程。UniAR 以预训练视觉编码器为基础,引入多层级特征融合机制与无需查表(lookup-free)的逐位量化(bitwise quantization)方案,在极低计算开销下,既保留高层语义信息,又兼顾底层细节,并有效扩展视觉词表规模。在此基础上,统一的自回归模型采用并行逐位预测(parallel-bitwise-prediction)策略,对空间上邻近、跨多个层级的视觉编码进行联合预测,显著缩短视觉序列长度,大幅提升生成速度。最后,基于扩散模型(diffusion-based)的视觉解码器直接作用于离散化视觉词元,重建高保真度图像。通过大规模预训练,辅以监督微调与强化学习优化,UniAR 在图像生成与图像编辑任务上达到当前最优性能,同时在多模态理解基准测试中亦保持强劲竞争力。项目主页详见:https://sharelab-sii.github.io/uniar-web。
Unified Multimodal Modeling aims to integrate visual understanding and generation within a single system. However, existing approaches typically rely on two disparate visual tokenizers, which splits the representation space and hinders truly unified modeling. We propose UniAR, a unified autoregressive framework where a single discrete visual tokenizer serves as the key bridge between understanding and generation, enabling a shared context in which the model can directly interpret its own generated visual tokens without additional re-encoding. UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme, preserving both high-level semantics and low-level details while scaling the effective visual vocabulary at minimal cost. Building on this, the unified autoregressive model adopts parallel-bitwise-prediction to jointly predict spatially grouped, multi-level visual codes, substantially reducing visual sequence length and accelerating generation. Finally, a diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images. Through large-scale pre-training, followed by supervised fine-tuning and reinforcement learning, UniAR achieves state-of-the-art performance on image generation and image editing while remaining competitive on multimodal understanding benchmarks. The project page is available at https://sharelab-sii.github.io/uniar-web.
许愿