MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

Xiang Yue ,

Tianyu Zheng ,

Yuansheng Ni ,

Yubo Wang ,

Kai Zhang ,

Shengbang Tong ,

Yuxuan Sun ,

Ming Yin ,

Botao Yu ,

Ge Zhang ,

Huan Sun ,

Yu Su ,

Wenhu Chen ,

Graham Neubig

Multimodal Intelligence VQA Visual CoT

本文介绍了MMMU-Pro，它是Massive Multi-discipline Multimodal Understanding and Reasoning（MMMU）基准测试的一个强健版本。MMM-Pro通过一个基于MMM的三步过程，严格评估多模态模型的真正理解和推理能力：（1）过滤出仅可由文本模型回答的问题，（2）增加候选选项，（3）引入一个仅包含图像的输入设置，其中问题嵌入在图像中。这种设置挑战AI真正实现“同时看到”和“同时阅读”，测试无缝集成视觉和文本信息的基本人类认知技能。结果显示，模型在MMM-Pro上的表现要比在MMM上低得多，跨模型范围从16.8％到26.9％不等。我们探讨了OCR提示和Chain of Thought（CoT）推理的影响，发现OCR提示的影响很小，而CoT通常可以提高性能。MMM-Pro提供了一个更严格的评估工具，密切模拟现实世界的情况，并为未来多模态AI研究提供了有价值的方向。

This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.

许愿