MMTBENCH: A Unified Benchmark for Complex Multimodal Table Reasoning

• URL: http://arxiv.org/abs/2505.21771v1
• Date: Tue, 27 May 2025 21:09:11 GMT
• Title: MMTBENCH: A Unified Benchmark for Complex Multimodal Table Reasoning
• Authors: Prasham Yatinkumar Titiya, Jainil Trivedi, Chitta Baral, Vivek Gupta,
• Abstract summary: MMTBENCH is a benchmark consisting of 500 real world multimodal tables drawn from diverse real world sources.<n>MMTBENCH questions cover four question types (Explicit, Implicit, Answer Mention, and Visual Based), five reasoning types (Mathematical, Extrema Identification, Fact Verification, Vision Based, and Others), and eight table types.
• Score: 40.95790862132066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal tables those that integrate semi structured data with visual elements such as charts and maps are ubiquitous across real world domains, yet they pose a formidable challenge to current vision language models (VLMs). While Large Language models (LLMs) and VLMs have demonstrated strong capabilities in text and image understanding, their performance on complex, real world multimodal table reasoning remains unexplored. To bridge this gap, we introduce MMTBENCH (Multimodal Table Benchmark), a benchmark consisting of 500 real world multimodal tables drawn from diverse real world sources, with a total of 4021 question answer pairs. MMTBENCH questions cover four question types (Explicit, Implicit, Answer Mention, and Visual Based), five reasoning types (Mathematical, Extrema Identification, Fact Verification, Vision Based, and Others), and eight table types (Single/Multiple Entity, Maps and Charts with Entities, Single/Multiple Charts, Maps, and Visualizations). Extensive evaluation of state of the art models on all types reveals substantial performance gaps, particularly on questions requiring visual-based reasoning and multi-step inference. These findings show the urgent need for improved architectures that more tightly integrate vision and language processing. By providing a challenging, high-quality resource that mirrors the complexity of real-world tasks, MMTBENCH underscores its value as a resource for future research on multimodal tables.

Related papers

• Efficient Table Retrieval and Understanding with Multimodal Large Language Models [22.49099892041409]
  Tabular data is frequently captured in image form across a wide range of real-world scenarios such as financial reports, handwritten records, and document scans.<n>These visual representations pose unique challenges for machine understanding, as they combine both structural and visual complexities.<n>We propose TabRAG, a framework that enables MLLMs to answer queries over large collections of table images.
  arXiv  Detail & Related papers  (2026-02-07T17:50:33Z)
• TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding [52.59372043981724]
  TableDART is a training-efficient framework that integrates multimodal views by reusing pretrained single-modality models.<n>In addition, we propose a novel agent to cross-modal knowledge integration by analyzing outputs from text- and image-based models.
  arXiv  Detail & Related papers  (2025-09-18T07:00:13Z)
• Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images [0.42970700836450476]
  Visual-TableQA is a large-scale, open-domain dataset designed to evaluate and enhance visual reasoning over complex data.<n>Visual-TableQA comprises 2.5k richly structured-rendered tables and 6k reasoning-intensive QA pairs, all produced at a cost of under USD 100.
  arXiv  Detail & Related papers  (2025-09-09T17:52:26Z)
• Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges [22.054723113358865]
  This paper introduces key concepts through a taxonomy of tabular input representations and an introduction of table understanding tasks.<n>Tables are two-dimensional, encompassing formats that range from well-structured database tables to complex, multi-layered spreadsheets, each with different purposes.<n>We highlight several critical gaps in the field that indicate the need for further research.
  arXiv  Detail & Related papers  (2025-07-31T23:41:31Z)
• Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports From Scratch with Agentic Framework [22.366142327629486]
  Multimodal DeepResearcher decomposes the task into four stages: researching, report textualization, planning, and multimodal report generation.<n>It achieves an 82% overall win rate over the baseline method.
  arXiv  Detail & Related papers  (2025-06-03T05:18:19Z)
• Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts [56.7225771305861]
  This paper introduces Multi-Modal Retrieval-Augmented Generation (M$2$RAG), a benchmark designed to evaluate the effectiveness of Multi-modal Large Language Models.<n>The benchmark comprises four tasks: image captioning, multi-modal question answering, multi-modal fact verification, and image reranking.<n>To enhance the context utilization capabilities of MLLMs, we also introduce Multi-Modal Retrieval-Augmented Instruction Tuning (MM-RAIT)
  arXiv  Detail & Related papers  (2025-02-24T16:25:25Z)
• Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy [2.294223504228228]
  Multimodal learning, a rapidly evolving field in artificial intelligence, seeks to construct more versatile and robust systems.<n>Inspired by the human ability to assimilate information through many senses, this method enables applications such as text-to-video conversion, visual question answering, and image captioning.<n>Recent developments in datasets that support multimodal language models (MLLMs) are highlighted in this overview.
  arXiv  Detail & Related papers  (2024-12-23T18:15:19Z)
• BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data [61.936320820180875]
  Large language models (LLMs) have become increasingly pivotal across various domains. BabelBench is an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement.
  arXiv  Detail & Related papers  (2024-10-01T15:11:24Z)
• Knowledge-Aware Reasoning over Multimodal Semi-structured Tables [85.24395216111462]
  This study investigates whether current AI models can perform knowledge-aware reasoning on multimodal structured data. We introduce MMTabQA, a new dataset designed for this purpose. Our experiments highlight substantial challenges for current AI models in effectively integrating and interpreting multiple text and image inputs.
  arXiv  Detail & Related papers  (2024-08-25T15:17:43Z)
• Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning [15.296263261737026]
  We introduce a Multi-Image MIRB Benchmark to evaluate visual language models' ability to compare, analyze, and reason across multiple images. Our benchmark encompasses four categories: perception, visual world knowledge, reasoning, and multi-hop reasoning. We demonstrate that while open-source VLMs were shown to approach the GPT-4V in single-image tasks, a significant gap remains in multi-image reasoning tasks.
  arXiv  Detail & Related papers  (2024-06-18T16:02:18Z)
• Multimodal Table Understanding [26.652797853893233]
  How to directly understand tables using intuitive visual information is a crucial and urgent challenge for developing more practical applications. We propose a new problem, multimodal table understanding, where the model needs to generate correct responses to various table-related requests. We develop Table-LLaVA, a generalist multimodal large language model (MLLM), which significantly outperforms recent open-source MLLM baselines on 23 benchmarks.
  arXiv  Detail & Related papers  (2024-06-12T11:27:03Z)
• TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy [81.76462101465354]
  We present a novel large vision-hugging model, TabPedia, equipped with a concept synergy mechanism. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such as table detection, table structure recognition, table querying, and table question answering. To better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA.
  arXiv  Detail & Related papers  (2024-06-03T13:54:05Z)
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning [40.972648044298374]
  Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. They often lack interpretability and struggle with complex visual inputs. We introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs. We propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts.
  arXiv  Detail & Related papers  (2024-03-25T17:59:23Z)
• NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision [64.83085920775316]
  We introduce NPHardEval4V, a multimodal benchmark suite grounded in four classical NP-hard problems.<n>Each task is presented through a combination of structured visual layouts and textual prompts, designed to assess the ability of LVLMs to perform reasoning under visual-linguistic constraints.<n>Our results show that while these models perform reasonably well on perception-based inputs, they struggle with global optimization, abstraction, and constraint satisfaction.
  arXiv  Detail & Related papers  (2024-03-04T07:10:31Z)
• Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion [112.27103169303184]
  Multimodal Knowledge Graphs (MKGs) organize visual-text factual knowledge. MKGformer can obtain SOTA performance on four datasets of multimodal link prediction, multimodal RE, and multimodal NER.
  arXiv  Detail & Related papers  (2022-05-04T23:40:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.