ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning

• URL: http://arxiv.org/abs/2608.13766v1
• Date: Thu, 13 Aug 2026 20:44:57 GMT
• Title: ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning
• Abstract summary: Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities.<n>We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on.<n>We introduce textbfChartProbe, a diagnostic framework whose probes are generated directly from the code that renders each chart.
• Score: 4.891649865325986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) remain unreliable on chart questions that require reasoning over visual quantities, and this weakness is usually attributed to a reasoning deficit and addressed with more reasoning supervision. We ask whether the difficulty lies in reasoning itself, or in the simpler skills that reasoning operates on: reading the plotted elements (\emph{perception}), locating them and binding them to their labels (\emph{grounding}), and performing single-step computations such as ranking, totals, and differences (\emph{simple reasoning}). We introduce \textbf{ChartProbe}, a diagnostic framework whose probes are generated directly from the code that renders each chart, so every gold answer is exact by construction, needs no human annotation, and attributes each failure to a single skill. ChartProbe enables an intervention prior work does not attempt: instead of synthesizing complex-reasoning data, we withhold complex questions and reasoning traces entirely, fine-tune on one simple skill at a time, and measure transfer to held-out complex-reasoning questions. Across three open-weight VLMs, supervising the simpler skills alone produces large gains on complex-reasoning questions the model never trained on: where these skills are weak and the model can be taught to read the image, training them recovers much of complex reasoning at no reasoning-data cost. The gains hold across three out-of-distribution settings: an unseen chart type (pie charts), a human-written benchmark disjoint from our images and templates (ChartQA), and a non-chart visual domain (CLEVR). Complex visual reasoning can therefore improve without complex-reasoning supervision.

Related papers

• CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning [58.582602613053204]
  Chart question answering (CQA) requires multimodal large language models (MLLMs) to integrate visual comprehension with logical reasoning.<n>Current MLLMs lack intrinsic visual grounded reasoning capabilities, leading to inaccurate perception and reasoning disconnected from visual evidence.<n>We propose CURV, a curriculum learning framework that develops intrinsic visual reasoning capabilities by reformulating CQA as multi-step visual grounded reasoning.
  arXiv  Detail & Related papers  (2026-08-03T19:48:00Z)
• Visual Graph Scaffolds for Structural Reasoning in Large Language Models [49.507575509154385]
  Graphs have been used to enhance large language models (LLMs)<n>In this paper, we take a different view: the value of graphs for LLMs lie not only in supplying information, but also in organizing reasoning.<n>Inspired by how humans use graph-structured mind maps to organize branching and converging thoughts, we ask whether graphs can serve as an internal form of reasoning assistance.
  arXiv  Detail & Related papers  (2026-06-01T12:17:52Z)
• DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams [54.39165467997251]
  We introduce DRAGON, a benchmark for evaluating evidence-grounded visual reasoning in diagrams.<n>Given a diagram, a question, and the correct answer, a model must predict bounding boxes that correspond to the visual elements required to justify the answer.<n>The DRAGON dataset contains 11,664 annotated question instances collected from six diagram QA datasets.
  arXiv  Detail & Related papers  (2026-04-28T05:24:05Z)
• ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch [57.01439313241121]
  We introduce Rollout Posterior Entropy (RPE), a novel metric that quantifies chart complexity.<n>We also develop truth-anchored inverse QA synthesis to guarantee reasoning rigor.<n>To further elevate difficulty and reasoning depth, we filter samples based on model fail-rate and distill high-quality Chain-of-Thought (CoT) reasoning.
  arXiv  Detail & Related papers  (2026-01-20T05:11:44Z)
• Visual Programmability: A Guide for Code-as-Thought in Chart Understanding [37.44645754630439]
  We propose a Code-as-Thought (CaT) approach to represent the visual information of a chart in a verifiable, symbolic format.<n>Visual Programmability is a learnable property that determines if a chart-question pair is better solved with code or direct visual analysis.<n>We implement this concept in an adaptive framework where a Vision-Language Models (VLMs) learns to choose between the CaT pathway and a direct visual reasoning pathway.
  arXiv  Detail & Related papers  (2025-09-11T09:22:16Z)
• RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning [63.599057862999]
  RefChartQA is a novel benchmark that integrates Chart Question Answering (ChartQA) with visual grounding.<n>Our experiments demonstrate that incorporating spatial awareness via grounding improves response accuracy by over 15%.
  arXiv  Detail & Related papers  (2025-03-29T15:50:08Z)
• VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning [13.011899331656018]
  VProChart is a novel framework designed to address the challenges of Chart Question Answering (CQA)<n>It integrates a lightweight Visual Perception Alignment Agent (VPAgent) and a Programmatic Solution Reasoning approach.<n>VProChart significantly outperforms existing methods, highlighting its capability in understanding and reasoning with charts.
  arXiv  Detail & Related papers  (2024-09-03T07:19:49Z)
• AltChart: Enhancing VLM-based Chart Summarization Through Multi-Pretext Tasks [31.414783623207477]
  We introduce the AltChart dataset, comprising 10,000 real chart images, each paired with a comprehensive summary. We propose a new method for pretraining Vision-Language Models (VLMs) to learn fine-grained chart representations. We conduct extensive evaluations of four leading chart summarization models, analyzing how accessible their descriptions are.
  arXiv  Detail & Related papers  (2024-05-22T12:18:52Z)
• CLEAR: Generative Counterfactual Explanations on Graphs [60.30009215290265]
  We study the problem of counterfactual explanation generation on graphs. A few studies have explored counterfactual explanations on graphs, but many challenges of this problem are still not well-addressed. We propose a novel framework CLEAR which aims to generate counterfactual explanations on graphs for graph-level prediction models.
  arXiv  Detail & Related papers  (2022-10-16T04:35:32Z)
• OpenCQA: Open-ended Question Answering with Charts [6.7038829115674945]
  We introduce a new task called OpenCQA, where the goal is to answer an open-ended question about a chart with texts. We implement and evaluate a set of baselines under three practical settings. Our analysis of the results show that the top performing models generally produce fluent and coherent text.
  arXiv  Detail & Related papers  (2022-10-12T23:37:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.