Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

• URL: http://arxiv.org/abs/2606.29808v1
• Date: Mon, 29 Jun 2026 05:40:35 GMT
• Title: Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework
• Abstract summary: Existing interactive tools are reliable but tedious, and mixed-initiative systems, while more efficient, lack generalizability.<n>Recent large language models (MLLMs) offer a unified interface for chart interpretation, yet their ability to extract accurate data tables, especially without visible labels, remains unclear.<n>We build a benchmark featuring diverse real-world charts without data labels to evaluate this capability.<n>Results show that, while current MLLMs reliably reconstruct table structures, they struggle with precise value recovery.
• Score: 27.45365558629541
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign. Existing interactive tools are reliable but tedious, and mixed-initiative systems, while more efficient, lack generalizability. Recent multimodal large language models (MLLMs) offer a unified interface for chart interpretation, yet their ability to extract accurate data tables, especially without visible labels, remains unclear. We build a benchmark featuring diverse real-world charts without data labels to evaluate this capability. Results show that, while current MLLMs reliably reconstruct table structures, they struggle with precise value recovery. To address this, we revisit chart data extraction from a human-centered perspective and argue that extraction should follow a progressive learning process similar to how people read charts. Our training framework substantially improves numerical accuracy, achieving state-of-the-art performance with a 7B-parameter model. A user study further shows that our model effectively supports mixed-initiative workflows for reliable chart data extraction.

Related papers

• DEEPCHART: How Far are LLMs from Faithful Data-Science Chart Generation? [54.230123199584305]
  We introduce DEEPCHART, an expert-annotated benchmark of 1,482 chart-generation instances from real-world scientific papers, financial filings, and ecosystem reports.<n> Experiments with state-of-the-art models show that visually plausible charts often conceal data-level hallucinations, with extraction and reasoning errors common in realistic long and multimodal settings.
  arXiv  Detail & Related papers  (2026-08-27T07:53:08Z)
• Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecture [50.95839069602553]
  Bar charts are commonly used in data visualization, and while they are easily understood by humans, it is non-trivial to extract the underlying data.<n>We present a per-bar, numerical value recovery pipeline for bar charts, where a JEPA encoder is used to produce semantically rich latent features.<n>The effectiveness of self-supervised finetuning and quality of the extracted features is evident when comparing our model to end-to-end supervised baselines.
  arXiv  Detail & Related papers  (2026-08-06T14:07:26Z)
• Self-Ensembling Vision-Language Models for Chart Data Extraction [0.0]
  manually digitizing charts is time-consuming and error-prone, motivating automatic chart-to-table extraction.<n>Recent approaches use specialized vision-language models (VLMs), yet performance still lags on charts with many datapoints or substantial stylistic variation.<n>We propose a VLM self-ensembling method that repeatedly samples multiple outputs from the same VLM for a fixed chart image and aggregates them at the level of individual table cells.
  arXiv  Detail & Related papers  (2026-05-26T17:10:51Z)
• ChartAB: A Benchmark for Chart Grounding & Dense Alignment [17.16234793106]
  We introduce a novel "ChartAlign Benchmark (ChartAB)" to provide a comprehensive evaluation of vision-language models (VLMs)<n>By incorporating a novel two-stage inference workflow, the benchmark can further evaluate VLMs capability to align and compare elements/attributes across two charts.<n>Our analysis of evaluations reveals new insights into their perception biases, weaknesses, robustness, and hallucinations in chart understanding.
  arXiv  Detail & Related papers  (2025-10-30T17:56:31Z)
• Are LLMs ready to help non-expert users to make charts of official statistics data? [39.88557897908524]
  We ask the question "Are current Generative AI models capable of facilitating the identification of the right data and the fully-automatic creation of charts to provide information in visual form?"<n>We present a structured evaluation of recent large language models' capabilities to generate charts from complex data in response to user queries.<n>Results indicate that locating and processing the correct data represents the most significant challenge.
  arXiv  Detail & Related papers  (2025-09-03T08:11:53Z)
• BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning [51.472854950300416]
  We propose BigCharts, a dataset creation pipeline that generates visually diverse chart images.<n>Unlike purely synthetic datasets, BigCharts incorporates real-world data, ensuring authenticity and visual diversity.<n>By introducing novel reward signals specifically designed for chart reasoning, our approach enhances model robustness and generalization.
  arXiv  Detail & Related papers  (2025-08-13T13:39:17Z)
• Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction Tuning [1.6570772838074355]
  multimodal large language models (MLLMs) exhibit great potential for chart question answering (CQA) Recent efforts primarily focus on scaling up training datasets through data collection and synthesis. We propose a visualization-referenced instruction tuning approach to guide the training dataset enhancement and model development.
  arXiv  Detail & Related papers  (2024-07-29T17:04:34Z)
• On Pre-training of Multimodal Language Models Customized for Chart Understanding [83.99377088129282]
  This paper explores the training processes necessary to improve MLLMs' comprehension of charts.<n>We introduce CHOPINLLM, an MLLM tailored for in-depth chart comprehension.
  arXiv  Detail & Related papers  (2024-07-19T17:58:36Z)
• ChartLlama: A Multimodal LLM for Chart Understanding and Generation [70.1393163657813]
  We create a high-quality instruction-tuning dataset leveraging GPT-4. Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset.
  arXiv  Detail & Related papers  (2023-11-27T15:20:23Z)
• StructChart: On the Schema, Metric, and Augmentation for Visual Chart Understanding [54.45681512355684]
  Current chart-related tasks focus on either chart perception that extracts information from the visual charts, or chart reasoning given the extracted data.<n>We introduce StructChart, a novel framework that leverages Structured Triplet Representations (STR) to achieve a unified and label-efficient approach.
  arXiv  Detail & Related papers  (2023-09-20T12:51:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.