Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy

• URL: http://arxiv.org/abs/2606.03142v1
• Date: Tue, 02 Jun 2026 04:36:32 GMT
• Title: Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy
• Abstract summary: Large Vision-Language Models (LVLMs) show strong visualization interpretation.<n>It is unclear whether their responses reflect genuine reasoning over visual evidence or factual priors learned during training.<n>We present a framework that isolates visual correctness from factual correctness.
• Score: 27.323597665236644
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) show strong visualization interpretation, yet it is unclear whether their responses reflect genuine reasoning over visual evidence or factual priors learned during training. Current evaluations mix these two sources, obscuring when correct visual interpretation is overridden by memorized facts. We present a framework that isolates visual correctness from factual correctness, revealing validity limitations in existing visualization literacy assessments. Across three experiments with 15 state-of-the-art LVLMs: (1) several models reach human-level performance on standard tests (VLAT), but this may reflect factual recall rather than visual understanding, while randomized-data tests (reVLAT) underestimate literacy when correct visual interpretation is superseded by factual priors. (2) Using our Counterfactual Visualization Literacy Assessment Test (CVLAT) with capability-normalized arbitration metrics, we classify models by the sign of their visual-factual reliance index (VFRI), revealing a visualization-oriented majority and a factual knowledge-oriented minority, though several near-zero cases warrant caution. A human baseline (N=30) on the same counterfactual items confirms that people overwhelmingly follow the chart under conflict, providing a human reference point. (3) Prompt-based intervention can shift prioritization, but its effectiveness is highly model-dependent and direction-asymmetric, and high chart-reading capability does not predict prompt-controllability. Overall, high visualization accuracy is not sufficient evidence of faithful visual reasoning: reliable integration into visual analytics requires evaluating not only visualization literacy but also how models arbitrate between visual evidence and factual priors when the two diverge. Benchmark and code: https://github.com/JaeyoungKim-HCIL/CVLAT

Related papers

• Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models [14.5102059776721]
  Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models.<n>However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge.<n>We explore these failures using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity.
  arXiv  Detail & Related papers  (2026-07-28T22:52:34Z)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models [66.95781712577315]
  We study the interplay between perception and reasoning in vision-language models (VLMs) post-training by decomposing their capabilities into three separate training stages.<n>We show that models trained with our approach achieve 1.5% higher reasoning accuracy with 20.8% shorter reasoning traces.
  arXiv  Detail & Related papers  (2026-05-19T17:58:40Z)
• CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models [44.585250047959924]
  Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors.<n>We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs.<n>We show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens.
  arXiv  Detail & Related papers  (2026-05-18T04:00:05Z)
• VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning [5.887179947363831]
  Large Vision Language Models (LVLMs) achieve strong multimodal reasoning but frequently exhibit hallucinations and incorrect responses with high certainty.<n>Existing verbalized confidence calibration methods, largely developed for text-only LLMs, typically optimize a single holistic confidence score using binary answer-level correctness.<n>We propose VL-Calibration, a reinforcement learning framework that explicitly decouples confidence into visual and reasoning confidence.
  arXiv  Detail & Related papers  (2026-04-10T17:47:19Z)
• Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities? [61.533560295383786]
  Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture.<n>We observe that U-MLLMs fail to maintain semantic equivalence when required to render the same results in the image modality.<n>We introduce VGUBench, a framework to decouple reasoning logic from generation fidelity.
  arXiv  Detail & Related papers  (2026-02-27T06:23:56Z)
• V-FAT: Benchmarking Visual Fidelity Against Text-bias [10.716447149075357]
  We investigate the tension between visual perception and linguistic priors.<n>We introduce V-FAT (Visual Fidelity Against Text-bias), a diagnostic benchmark comprising 4,026 VQA instances across six semantic domains.<n>Our evaluation of 12 frontier MLLMs reveals that while models excel in existing benchmarks, they experience significant visual collapse under high linguistic dominance.
  arXiv  Detail & Related papers  (2026-01-08T12:50:14Z)
• BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception [67.89135437537179]
  We introduce BLINK-Twice, a vision-centric reasoning benchmark grounded in challenging perceptual tasks.<n>Instead of relying on external knowledge, our tasks require models to reason from visual content alone.<n>Compared to prior perception benchmarks, it moves beyond shallow perception and requires fine-grained observation and analytical reasoning.
  arXiv  Detail & Related papers  (2025-10-10T13:14:13Z)
• Self-Rewarding Vision-Language Model via Reasoning Decomposition [49.784411666601905]
  Vision-Language Models (VLMs) often suffer from visual hallucinations, saying things that are not actually in the image, and language shortcuts.<n>We introduce Vision-SR1, a self-rewarding method that improves visual reasoning without relying on external visual supervisions.<n>Our experiments demonstrate that Vision-SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts.
  arXiv  Detail & Related papers  (2025-08-27T08:01:03Z)
• Tell Me Without Telling Me: Two-Way Prediction of Visualization Literacy and Visual Attention [20.87674676293376]
  We show that distinct attention patterns in visual data exploration can correlate with participants' literacy levels.<n>We propose two computational models leveraging these insights: Lit2Sal and Sal2Lit.
  arXiv  Detail & Related papers  (2025-07-22T20:18:45Z)
• FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability [10.184567639685321]
  We introduce FiVL, a novel method for constructing datasets designed to train LVLMs for enhanced visual grounding.<n>We present benchmarks to assess the model's ability to use image as substantive evidence.<n>We identify attention heads with the strongest vision-language alignment, enabling explainability on visual-driven hallucinations.
  arXiv  Detail & Related papers  (2024-12-19T09:24:10Z)
• Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs [52.497823009176074]
  Large Vision-Language Models (LVLMs) often produce responses that misalign with factual information, a phenomenon known as hallucinations.<n>We introduce Visual Description Grounded Decoding (VDGD), a training-free method designed to enhance visual perception and improve reasoning capabilities in LVLMs.
  arXiv  Detail & Related papers  (2024-05-24T16:21:59Z)
• Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models [57.08925810659545]
  We conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models. Our empirical observations suggest that vision-and-language models are better at label prediction tasks. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
  arXiv  Detail & Related papers  (2022-12-01T05:00:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.