Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization

• URL: http://arxiv.org/abs/2312.10610v1
• Date: Sun, 17 Dec 2023 05:13:58 GMT
• Title: Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization
• Authors: Xuan Long Do, Mohammad Hassanpour, Ahmed Masry, Parsa Kavehzadeh, Enamul Hoque, Shafiq Joty
• Abstract summary: Large language models (LLMs) have shown impressive generalization capabilities to unseen tasks. We propose PromptChart, a multimodal few-shot prompting framework with LLMs for chart-related applications. Our experiments on three different chart-related information consumption tasks show that with properly designed prompts LLMs can excel on the benchmarks.
• Score: 27.913656283822483
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A number of tasks have been proposed recently to facilitate easy access to charts such as chart QA and summarization. The dominant paradigm to solve these tasks has been to fine-tune a pretrained model on the task data. However, this approach is not only expensive but also not generalizable to unseen tasks. On the other hand, large language models (LLMs) have shown impressive generalization capabilities to unseen tasks with zero- or few-shot prompting. However, their application to chart-related tasks is not trivial as these tasks typically involve considering not only the underlying data but also the visual features in the chart image. We propose PromptChart, a multimodal few-shot prompting framework with LLMs for chart-related applications. By analyzing the tasks carefully, we have come up with a set of prompting guidelines for each task to elicit the best few-shot performance from LLMs. We further propose a strategy to inject visual information into the prompts. Our experiments on three different chart-related information consumption tasks show that with properly designed prompts LLMs can excel on the benchmarks, achieving state-of-the-art.

Related papers

• Distill Visual Chart Reasoning Ability from LLMs to MLLMs [38.62832112530892]
  Solving complex chart Q&A tasks requires advanced visual reasoning abilities in multimodal large language models (MLLMs) We propose Code-as-Intermediary Translation (CIT), a cost-effective, efficient and easily scalable data synthesis method for distilling visual reasoning abilities from LLMs to MLLMs. We employ text-based synthesizing techniques to construct chart-plotting code and produce ReachQA, a dataset containing 3k reasoning-intensive charts and 20k Q&A pairs.
  arXiv  Detail & Related papers  (2024-10-24T14:50:42Z)
• MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems [18.188725200923333]
  Existing benchmarks for chart-related tasks fall short in capturing the complexity of real-world multi-chart scenarios. We introduce MultiChartQA, a benchmark that evaluates MLLMs' capabilities in four key areas: direct question answering, parallel question answering, comparative reasoning, and sequential reasoning. Our results highlight the challenges in multi-chart comprehension and the potential of MultiChartQA to drive advancements in this field.
  arXiv  Detail & Related papers  (2024-10-18T05:15:50Z)
• Enhance Graph Alignment for Large Language Models [33.96082485852042]
  Graph-to-token approaches are popular in enabling Large Language Models to process graph information. Existing methods have a misalignment between self-supervised tasks and supervised downstream tasks. We propose Graph Alignment Large Language Models (GALLM) to benefit from aligned task templates.
  arXiv  Detail & Related papers  (2024-10-15T07:50:34Z)
• AltChart: Enhancing VLM-based Chart Summarization Through Multi-Pretext Tasks [31.414783623207477]
  We introduce the AltChart dataset, comprising 10,000 real chart images, each paired with a comprehensive summary. We propose a new method for pretraining Vision-Language Models (VLMs) to learn fine-grained chart representations. We conduct extensive evaluations of four leading chart summarization models, analyzing how accessible their descriptions are.
  arXiv  Detail & Related papers  (2024-05-22T12:18:52Z)
• TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning [83.58521787193293]
  We present TinyChart, an efficient MLLM for chart understanding with only 3B parameters. TinyChart overcomes two key challenges in efficient chart understanding: (1) reduce the burden of learning numerical computations through a Program-of-Thoughts (PoT) learning strategy, and (2) reduce lengthy vision feature sequences produced by the vision transformer for high-resolution images through a Vision Token Merging module.
  arXiv  Detail & Related papers  (2024-04-25T14:23:24Z)
• ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning [28.204261069650897]
  We introduce ChartInstruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. In experiments on four downstream tasks, we first show the effectiveness of our model--achieving a new set of state-of-the-art results.
  arXiv  Detail & Related papers  (2024-03-14T01:40:23Z)
• ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning [54.89249749894061]
  ChartAssistant is a vision-language model for universal chart comprehension and reasoning. It undergoes a two-stage training process, starting with pre-training on chart-to-table parsing to align chart and text. Experimental results demonstrate significant performance gains over the state-of-the-art UniChart and Chartllama method.
  arXiv  Detail & Related papers  (2024-01-04T17:51:48Z)
• StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding [58.38480335579541]
  Current chart-related tasks focus on either chart perception which refers to extracting information from the visual charts, or performing reasoning given the extracted data. In this paper, we aim to establish a unified and label-efficient learning paradigm for joint perception and reasoning tasks. Experiments are conducted on various chart-related tasks, demonstrating the effectiveness and promising potential for a unified chart perception-reasoning paradigm.
  arXiv  Detail & Related papers  (2023-09-20T12:51:13Z)
• ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules [89.75395046894809]
  We present ChartReader, a unified framework that seamlessly integrates chart derendering and comprehension tasks. Our approach includes a transformer-based chart component detection module and an extended pre-trained vision-language model for chart-to-X tasks. Our proposed framework can significantly reduce the manual effort involved in chart analysis, providing a step towards a universal chart understanding model.
  arXiv  Detail & Related papers  (2023-04-05T00:25:27Z)
• Unsupervised Task Graph Generation from Instructional Video Transcripts [53.54435048879365]
  We consider a setting where text transcripts of instructional videos performing a real-world activity are provided. The goal is to identify the key steps relevant to the task as well as the dependency relationship between these key steps. We propose a novel task graph generation approach that combines the reasoning capabilities of instruction-tuned language models along with clustering and ranking components.
  arXiv  Detail & Related papers  (2023-02-17T22:50:08Z)
• PartAL: Efficient Partial Active Learning in Multi-Task Visual Settings [57.08386016411536]
  We show that it is more effective to select not only the images to be annotated but also a subset of tasks for which to provide annotations at each Active Learning (AL) We demonstrate the effectiveness of our approach on several popular multi-task datasets.
  arXiv  Detail & Related papers  (2022-11-21T15:08:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.