Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning

• URL: http://arxiv.org/abs/2312.10160v2
• Date: Thu, 30 May 2024 05:27:35 GMT
• Title: Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning
• Authors: Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R. Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, Heng Ji,
• Abstract summary: We introduce a comprehensive typology of factual errors in generated chart captions. A large-scale human annotation effort provides insight into the error patterns and frequencies in captions crafted by various chart captioning models. Our analysis reveals that even state-of-the-art models, including GPT-4V, frequently produce captions laced with factual inaccuracies.
• Score: 90.13978453378768
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in large vision-language models (LVLMs) have led to significant progress in generating natural language descriptions for visual content and thus enhancing various applications. One issue with these powerful models is that they sometimes produce texts that are factually inconsistent with the visual input. While there has been some effort to mitigate such inconsistencies in natural image captioning, the factuality of generated captions for structured document images, such as charts, has not received as much scrutiny, posing a potential threat to information reliability in critical applications. This work delves into the factuality aspect by introducing a comprehensive typology of factual errors in generated chart captions. A large-scale human annotation effort provides insight into the error patterns and frequencies in captions crafted by various chart captioning models, ultimately forming the foundation of a novel dataset, CHOCOLATE. Our analysis reveals that even state-of-the-art models, including GPT-4V, frequently produce captions laced with factual inaccuracies. In response to this challenge, we establish the new task of Chart Caption Factual Error Correction and introduce CHARTVE, a model for visual entailment that outperforms proprietary and open-source LVLMs in evaluating factual consistency. Furthermore, we propose C2TFEC, an interpretable two-stage framework that excels at correcting factual errors. This work inaugurates a new domain in factual error correction for chart captions, presenting a novel evaluation mechanism, and demonstrating an effective approach to ensuring the factuality of generated chart captions. The code and data as well as the continuously updated benchmark can be found at: https://khuangaf.github.io/CHOCOLATE/.

Related papers

• PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions [55.95282725491425]
  PoSh is a metric for detailed image description that uses scene graphs as structured rubrics to guide LLMs-as-a-Judge.<n>PoSh is replicable, interpretable and a better proxy for human raters than existing metrics.<n>We show that PoSh achieves stronger correlations with the human judgments in DOCENT than the best open-weight alternatives.
  arXiv  Detail & Related papers  (2025-10-21T20:30:20Z)
• Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models [25.564425023762045]
  Current vision-language models (VLMs) still struggle on basic data visualization understanding tasks.<n>Are VLM failures attributable to limitations in how visual information in the data visualization is encoded, how information is transferred between the vision and language modules, or how information is processed within the language module?<n>We developed FUGU, a suite of data visualization understanding tasks, to precisely characterize potential sources of difficulty.
  arXiv  Detail & Related papers  (2025-10-02T18:29:07Z)
• Vision Language Models Are Not (Yet) Spelling Correctors [0.742779257315787]
  Spelling correction from visual input poses unique challenges for vision language models (VLMs)<n>We present ReViCo, the first benchmark that systematically evaluates VLMs on real-world visual spelling correction across Chinese and English.
  arXiv  Detail & Related papers  (2025-09-22T07:10:42Z)
• BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning [51.472854950300416]
  We propose BigCharts, a dataset creation pipeline that generates visually diverse chart images.<n>Unlike purely synthetic datasets, BigCharts incorporates real-world data, ensuring authenticity and visual diversity.<n>By introducing novel reward signals specifically designed for chart reasoning, our approach enhances model robustness and generalization.
  arXiv  Detail & Related papers  (2025-08-13T13:39:17Z)
• ChartCap: Mitigating Hallucination of Dense Chart Captioning [37.96805802388932]
  We introduce ChartCap, a large-scale dataset of 565K real-world chart images paired with type-specific, dense captions.<n>To build ChartCap, we design a four-stage pipeline that generates captions using only the discernible data from the chart.<n>We propose a novel metric, the Visual Consistency Score, which evaluates caption quality by measuring the similarity between the chart regenerated from a caption and the original chart.
  arXiv  Detail & Related papers  (2025-08-05T07:09:07Z)
• On Pre-training of Multimodal Language Models Customized for Chart Understanding [83.99377088129282]
  This paper explores the training processes necessary to improve MLLMs' comprehension of charts. We introduce CHOPINLLM, an MLLM tailored for in-depth chart comprehension.
  arXiv  Detail & Related papers  (2024-07-19T17:58:36Z)
• Text Data-Centric Image Captioning with Interactive Prompts [20.48013600818985]
  Supervised image captioning approaches have made great progress, but it is challenging to collect high-quality human-annotated image-text data. This paper proposes a new Text data-centric approach with Interactive Prompts for image Captioning, named TIPCap.
  arXiv  Detail & Related papers  (2024-03-28T07:43:49Z)
• Is it an i or an l: Test-time Adaptation of Text Line Recognition Models [9.149602257966917]
  We introduce the problem of adapting text line recognition models during test time. We propose an iterative self-training approach that uses feedback from the language model to update the optical model. Experimental results show that the proposed adaptation method offers an absolute improvement of up to 8% in character error rate.
  arXiv  Detail & Related papers  (2023-08-29T05:44:00Z)
• You Can Generate It Again: Data-to-Text Generation with Verification and Correction Prompting [24.738004421537926]
  Small language models like T5 excel in generating high-quality text for data-to-text tasks.<n>They frequently miss keywords, which is considered one of the most severe and common errors in this task.<n>We explore the potential of using feedback systems to enhance semantic fidelity in smaller language models for data-to-text generation tasks.
  arXiv  Detail & Related papers  (2023-06-28T05:34:25Z)
• FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing [66.70054075041487]
  Existing scene graphs that convert image captions into scene graphs often suffer from two types of errors. First, the generated scene graphs fail to capture the true semantics of the captions or the corresponding images, resulting in a lack of faithfulness. Second, the generated scene graphs have high inconsistency, with the same semantics represented by different annotations.
  arXiv  Detail & Related papers  (2023-05-27T15:38:31Z)
• Models See Hallucinations: Evaluating the Factuality in Video Captioning [57.85548187177109]
  We conduct a human evaluation of the factuality in video captioning and collect two annotated factuality datasets. We find that 57.0% of the model-generated sentences have factual errors, indicating it is a severe problem in this field. We propose a weakly-supervised, model-based factuality metric FactVC, which outperforms previous metrics on factuality evaluation of video captioning.
  arXiv  Detail & Related papers  (2023-03-06T08:32:50Z)
• Towards Fine-Grained Information: Identifying the Type and Location of Translation Errors [80.22825549235556]
  Existing approaches can not synchronously consider error position and type. We build an FG-TED model to predict the textbf addition and textbfomission errors. Experiments show that our model can identify both error type and position concurrently, and gives state-of-the-art results.
  arXiv  Detail & Related papers  (2023-02-17T16:20:33Z)
• FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations [114.94628499698096]
  We propose FactGraph, a method that decomposes the document and the summary into structured meaning representations (MRs) MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form, and reducing data sparsity. Experiments on different benchmarks for evaluating factuality show that FactGraph outperforms previous approaches by up to 15%.
  arXiv  Detail & Related papers  (2022-04-13T16:45:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.