Enhancing Question Answering on Charts Through Effective Pre-training Tasks

• URL: http://arxiv.org/abs/2406.10085v2
• Date: Fri, 04 Oct 2024 01:33:23 GMT
• Title: Enhancing Question Answering on Charts Through Effective Pre-training Tasks
• Authors: Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang, Shalin Shah,
• Abstract summary: We address the limitation of current VisualQA models when applied to charts and plots. Our findings indicate that existing models particularly underperform in answering questions related to the chart's structural and visual context. We propose three simple pre-training tasks that enforce the existing model in terms of both structural-visual knowledge, as well as its understanding of numerical questions.
• Score: 26.571522748519584
• License:
• Abstract: To completely understand a document, the use of textual information is not enough. Understanding visual cues, such as layouts and charts, is also required. While the current state-of-the-art approaches for document understanding (both OCR-based and OCR-free) work well, a thorough analysis of their capabilities and limitations has not yet been performed. Therefore, in this work, we addresses the limitation of current VisualQA models when applied to charts and plots. To investigate shortcomings of the state-of-the-art models, we conduct a comprehensive behavioral analysis, using ChartQA as a case study. Our findings indicate that existing models particularly underperform in answering questions related to the chart's structural and visual context, as well as numerical information. To address these issues, we propose three simple pre-training tasks that enforce the existing model in terms of both structural-visual knowledge, as well as its understanding of numerical questions. We evaluate our pre-trained model (called MatCha-v2) on three chart datasets - both extractive and abstractive question datasets - and observe that it achieves an average improvement of 1.7% over the baseline model.

Related papers

• RealCQA-V2 : Visual Premise Proving A Manual COT Dataset for Charts [2.9201864249313383]
  We introduce Visual Premise Proving, a novel task tailored to refine the process of chart question answering. This approach represents a departure from conventional accuracy-based evaluation methods. A model adept at reasoning is expected to demonstrate proficiency in both data retrieval and the structural understanding of charts.
  arXiv  Detail & Related papers  (2024-10-29T19:32:53Z)
• On Pre-training of Multimodal Language Models Customized for Chart Understanding [83.99377088129282]
  This paper explores the training processes necessary to improve MLLMs' comprehension of charts. We introduce CHOPINLLM, an MLLM tailored for in-depth chart comprehension.
  arXiv  Detail & Related papers  (2024-07-19T17:58:36Z)
• Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness [47.68358935792437]
  Chart question answering (CQA) is a crucial area of Visual Language Understanding. Current Visual Language Models (VLMs) in this field remain under-explored. This paper evaluates state-of-the-art VLMs on comprehensive datasets.
  arXiv  Detail & Related papers  (2024-07-15T20:29:24Z)
• From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models [98.41645229835493]
  Data visualization in the form of charts plays a pivotal role in data analysis, offering critical insights and aiding in informed decision-making. Large foundation models, such as large language models, have revolutionized various natural language processing tasks. This survey paper serves as a comprehensive resource for researchers and practitioners in the fields of natural language processing, computer vision, and data analysis.
  arXiv  Detail & Related papers  (2024-03-18T17:57:09Z)
• ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning [28.204261069650897]
  We introduce ChartInstruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. In experiments on four downstream tasks, we first show the effectiveness of our model--achieving a new set of state-of-the-art results.
  arXiv  Detail & Related papers  (2024-03-14T01:40:23Z)
• StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding [58.38480335579541]
  Current chart-related tasks focus on either chart perception which refers to extracting information from the visual charts, or performing reasoning given the extracted data. In this paper, we aim to establish a unified and label-efficient learning paradigm for joint perception and reasoning tasks. Experiments are conducted on various chart-related tasks, demonstrating the effectiveness and promising potential for a unified chart perception-reasoning paradigm.
  arXiv  Detail & Related papers  (2023-09-20T12:51:13Z)
• RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order Logic [8.155575318208628]
  We introduce a benchmark and dataset for chart visual QA on real-world charts. Our contribution includes the introduction of a new answer type, 'list', with both ranked and unranked variations. Results of our experiments, conducted on a real-world out-of-distribution dataset, provide a robust evaluation of large-scale pre-trained models.
  arXiv  Detail & Related papers  (2023-08-03T18:21:38Z)
• OpenCQA: Open-ended Question Answering with Charts [6.7038829115674945]
  We introduce a new task called OpenCQA, where the goal is to answer an open-ended question about a chart with texts. We implement and evaluate a set of baselines under three practical settings. Our analysis of the results show that the top performing models generally produce fluent and coherent text.
  arXiv  Detail & Related papers  (2022-10-12T23:37:30Z)
• Towards Complex Document Understanding By Discrete Reasoning [77.91722463958743]
  Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language. We introduce a new Document VQA dataset, named TAT-DQA, which consists of 3,067 document pages and 16,558 question-answer pairs. We develop a novel model named MHST that takes into account the information in multi-modalities, including text, layout and visual image, to intelligently address different types of questions.
  arXiv  Detail & Related papers  (2022-07-25T01:43:19Z)
• Question-Answer Sentence Graph for Joint Modeling Answer Selection [122.29142965960138]
  We train and integrate state-of-the-art (SOTA) models for computing scores between question-question, question-answer, and answer-answer pairs. Online inference is then performed to solve the AS2 task on unseen queries.
  arXiv  Detail & Related papers  (2022-02-16T05:59:53Z)
• Classification-Regression for Chart Comprehension [16.311371103939205]
  Chart question answering (CQA) is a task used for assessing chart comprehension. We propose a new model that jointly learns classification and regression. Our model's edge is particularly emphasized on questions with out-of-vocabulary answers.
  arXiv  Detail & Related papers  (2021-11-29T18:46:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.