Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

Shravan Venkatraman ,

Ritesh Thawkar ,

Omkar Thawakar ,

Rao Muhammad Anwer ,

Hisham Cholakkal ,

Salman Khan ,

Fahad Khan

Multimodal Intelligence Vision-Language Pre-training VQA MHD Agent Autonomous Workflows

2026年06月25日

近期，能够自主演化的大型多模态模型（LMM）因其可在纯无监督场景下提升视觉推理能力而受到广泛关注。然而，现有自主演化型LMM所采用的多角色自博弈（multi-role self-play）与自一致性奖励机制，仅以答案层面的一致性为目标进行优化，却未能确保解码器真正关注视觉内容；相反，模型主要依赖统计性语言先验（statistical language priors）来生成在语言层面自洽的输出。这导致了一种持续存在的失效模式，我们将其命名为“视觉条件不足”（visual under-conditioning）：即解码器在生成过程中过度依赖语言先验，而非图像本身，具体表现为对视觉标记（visual tokens）的关注显著不足。正因如此，当前的自主演化型LMM在图像描述生成（image captioning）和视觉问答（visual question answering）等视觉–语言理解任务上表现乏力。为应对这一问题，我们提出VISE（Visual Invariance Self-Evolution，基于视觉不变性的自主演化框架）——一种纯无监督的自主演化方法，其核心在于通过两种互补的、基于不变性（invariance-based）的奖励信号，直接约束并规范模型对视觉信息的依赖策略：一是几何不变性奖励（geometric invariance reward），要求模型在已知几何变换（如旋转、缩放、裁剪等）下保持空间表征的一致性；二是语义不变性奖励（semantic invariance reward），通过扰动模型预测的关键图像区域并要求其识别证据缺失，从而惩罚脱离视觉证据的盲目生成行为。VISE完全运行于单一模型内部，无需设置专用角色（specialist roles）、外部奖励模型（external reward models）或任何人工标注，且仅需原始未标注图像即可完成训练。我们在18个基准数据集上的实验充分验证了该方法的有效性：以Qwen3-VL-2B为基座模型时，VISE在COCO数据集上CIDEr指标提升达+16.85，在TextCaps数据集上提升达+19.66；同时将物体幻觉（object hallucination）程度降低5.0个CHAIR-I得分；此外，该方法展现出优异的泛化能力，可跨四种不同模型家族及多个参数规模稳定生效。我们的代码与预训练模型已开源，详见：https://mbzuai-oryx.github.io/VISE

Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision--language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18 benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE

许愿