Multimodal Table Understanding

Mingyu Zheng ,

Xinwei Feng ,

Qingyi Si ,

Qiaoqiao She ,

Zheng Lin ,

Wenbin Jiang ,

Weiping Wang

LLM PTSAD SFT Multimodal Intelligence Vision-Language Pre-training VRDU

虽然之前的表格理解方法已经取得了很大的进展，包括基于大型语言模型（LLMs）的近期方法，但它们严重依赖于这样的前提：给定的表格必须被转换成某种文本序列（如Markdown或HTML）以作为模型输入。然而，在某些实际场景下很难获得这样高质量的文本表格表示，而表格图像则更加易于获取。因此，如何直接使用直观的视觉信息来理解表格是开发更实用应用的一个关键和紧迫的挑战。在本文中，我们提出了一个新的问题，即多模态表格理解，其中模型需要根据给定的表格图像对各种与表格相关的请求生成正确的响应。为了促进模型的训练和评估，我们构建了一个名为MMTab的大规模数据集，它涵盖了广泛的表格图像、指令和任务。在此基础上，我们开发了一个名为Table-LLaVA的通用表格多模态大型语言模型（MLLM），在23个基准测试中，在保留和未保留的设置下，显著优于最近的开源MLLM基线。代码和数据可在以下网址找到：https://github.com/SpursGoZmy/Table-LLaVA。

Although great progress has been made by previous table understanding methods including recent approaches based on large language models (LLMs), they rely heavily on the premise that given tables must be converted into a certain text sequence (such as Markdown or HTML) to serve as model input. However, it is difficult to access such high-quality textual table representations in some real-world scenarios, and table images are much more accessible. Therefore, how to directly understand tables using intuitive visual information is a crucial and urgent challenge for developing more practical applications. In this paper, we propose a new problem, multimodal table understanding, where the model needs to generate correct responses to various table-related requests based on the given table image. To facilitate both the model training and evaluation, we construct a large-scale dataset named MMTab, which covers a wide spectrum of table images, instructions and tasks. On this basis, we develop Table-LLaVA, a generalist tabular multimodal large language model (MLLM), which significantly outperforms recent open-source MLLM baselines on 23 benchmarks under held-in and held-out settings. The code and data is available at this https://github.com/SpursGoZmy/Table-LLaVA

许愿