Self-Ensembling Vision-Language Models for Chart Data Extraction

• URL: http://arxiv.org/abs/2605.27298v1
• Date: Tue, 26 May 2026 17:10:51 GMT
• Title: Self-Ensembling Vision-Language Models for Chart Data Extraction
• Abstract summary: manually digitizing charts is time-consuming and error-prone, motivating automatic chart-to-table extraction.<n>Recent approaches use specialized vision-language models (VLMs), yet performance still lags on charts with many datapoints or substantial stylistic variation.<n>We propose a VLM self-ensembling method that repeatedly samples multiple outputs from the same VLM for a fixed chart image and aggregates them at the level of individual table cells.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Charts effectively convey quantitative information, but the underlying data are often locked in image form, hindering reuse and analysis. Manually digitizing charts is time-consuming and error-prone, motivating automatic chart-to-table extraction. Recent approaches use specialized vision-language models (VLMs), yet performance still lags on charts with many datapoints or substantial stylistic variation. We propose a VLM self-ensembling method that repeatedly samples multiple tabular outputs from the same VLM for a fixed chart image and aggregates them at the level of individual table cells. We align candidate tables and take per-cell medians over numerical values to produce a more accurate consensus table. Our method also includes convergence detection to stop sampling once the aggregated table stabilizes, and uncertainty estimation based on dispersion across samples to help users assess extraction reliability. Because existing chart extraction benchmarks contain relatively simple plots with limited room for improvement, we introduce WB-ChartExtract, a new benchmark built from World Bank data with more complex and stylistically diverse charts; on average, its charts contain 7 times more datapoints than those in the ChartQA benchmark. Across both ChartQA and WB-ChartExtract, our approach improves extraction accuracy over single-pass VLM outputs, yielding up to 23% relative improvement on WB-ChartExtract after ensembling. More broadly, our method helps unlock tabular data previously siloed in chart images, enabling downstream analysis and reuse.

Related papers

• What You See Is What You Get: Observation-Aligned Supervision for Chart-to-Code Generation [50.62320148925983]
  Many chart programs contain latent raw variables that cannot be uniquely recovered from the rendered image.<n>We introduce Observation-Aligned supervision, a rewriting framework that replaces latent raw-data targets with quantities constrained by the visual observation.<n>Our results suggest that improving chart-to-code models requires not only more data or advanced learning objectives or algorithms, but also supervision targets that respect what is identifiable from the chart image.
  arXiv  Detail & Related papers  (2026-07-06T06:53:40Z)
• Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework [27.45365558629541]
  Existing interactive tools are reliable but tedious, and mixed-initiative systems, while more efficient, lack generalizability.<n>Recent large language models (MLLMs) offer a unified interface for chart interpretation, yet their ability to extract accurate data tables, especially without visible labels, remains unclear.<n>We build a benchmark featuring diverse real-world charts without data labels to evaluate this capability.<n>Results show that, while current MLLMs reliably reconstruct table structures, they struggle with precise value recovery.
  arXiv  Detail & Related papers  (2026-06-29T05:40:35Z)
• ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation [51.49421299447412]
  Multimodal large language models (MLLMs) are increasingly used to automate chart generation from data tables.<n>We introduce ChartAttack, a framework for evaluating how MLLMs can be misused to generate misleading charts at scale.
  arXiv  Detail & Related papers  (2026-01-19T11:57:48Z)
• ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning [54.86473583610112]
  We propose PointCoT, which integrates reflective interaction into chain-of-thought reasoning in charts.<n>By prompting MLLMs to generate bounding boxes and re-render charts based on location annotations, we establish connections between textual reasoning steps and visual grounding regions.<n>We develop two instruction-tuned models, ChartPointQ2 and ChartPointQ2.5, which outperform state-of-the-art across several chart benchmarks.
  arXiv  Detail & Related papers  (2025-11-29T04:01:55Z)
• BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning [51.472854950300416]
  We propose BigCharts, a dataset creation pipeline that generates visually diverse chart images.<n>Unlike purely synthetic datasets, BigCharts incorporates real-world data, ensuring authenticity and visual diversity.<n>By introducing novel reward signals specifically designed for chart reasoning, our approach enhances model robustness and generalization.
  arXiv  Detail & Related papers  (2025-08-13T13:39:17Z)
• End-to-End Chart Summarization via Visual Chain-of-Thought in Vision-Language Models [0.0]
  This paper introduces End-to-End Visual Chain-of-Thought (V-CoT) for chart summarization.<n>Our method directly trains an LVLM to process chart images and generate textual summaries in an end-to-end fashion.<n>We incorporate a visual Chain-of-Thought mechanism through instruction fine-tuning, implicitly guiding the LVLM to perform visual reasoning steps.
  arXiv  Detail & Related papers  (2025-02-24T19:13:45Z)
• OneChart: Purify the Chart Structural Extraction via One Auxiliary Token [23.271325787765875]
  OneChart is a reliable agent specifically devised for the structural extraction of chart information. We introduce an auxiliary token placed at the beginning of the total tokens along with an additional decoder. With the aid of the auxiliary token, we have devised a self-evaluation mechanism that enables the model to gauge the reliability of its chart parsing results.
  arXiv  Detail & Related papers  (2024-04-15T17:58:57Z)
• ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning [55.22996841790139]
  We benchmark the ability of off-the-shelf Multi-modal Large Language Models (MLLMs) in the chart domain.<n>We construct ChartX, a multi-modal evaluation set covering 18 chart types, 7 chart tasks, 22 disciplinary topics, and high-quality chart data.<n>We develop ChartVLM to offer a new perspective on handling multi-modal tasks that strongly depend on interpretable patterns.
  arXiv  Detail & Related papers  (2024-02-19T14:48:23Z)
• ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules [89.75395046894809]
  We present ChartReader, a unified framework that seamlessly integrates chart derendering and comprehension tasks. Our approach includes a transformer-based chart component detection module and an extended pre-trained vision-language model for chart-to-X tasks. Our proposed framework can significantly reduce the manual effort involved in chart analysis, providing a step towards a universal chart understanding model.
  arXiv  Detail & Related papers  (2023-04-05T00:25:27Z)
• Tensor Fields for Data Extraction from Chart Images: Bar Charts and Scatter Plots [0.0]
  Automated chart reading involves data extraction and contextual understanding of the data from chart images. We identify an appropriate tensor field as the model and propose a methodology for the use of its degenerate point extraction for data extraction from chart images. Our results show that tensor voting is effective for data extraction from bar charts and scatter plots, and histograms, as a special case of bar charts.
  arXiv  Detail & Related papers  (2020-10-05T20:19:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.