Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model

• URL: http://arxiv.org/abs/2010.09142v2
• Date: Sun, 29 Nov 2020 21:17:49 GMT
• Title: Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model
• Authors: Jason Obeid and Enamul Hoque
• Abstract summary: We introduce a new dataset and present a neural model for automatically generating natural language summaries for charts. The generated summaries provide an interpretation of the chart and convey the key insights found within that chart.
• Score: 6.320141734801679
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Information visualizations such as bar charts and line charts are very popular for exploring data and communicating insights. Interpreting and making sense of such visualizations can be challenging for some people, such as those who are visually impaired or have low visualization literacy. In this work, we introduce a new dataset and present a neural model for automatically generating natural language summaries for charts. The generated summaries provide an interpretation of the chart and convey the key insights found within that chart. Our neural model is developed by extending the state-of-the-art model for the data-to-text generation task, which utilizes a transformer-based encoder-decoder architecture. We found that our approach outperforms the base model on a content selection metric by a wide margin (55.42% vs. 8.49%) and generates more informative, concise, and coherent summaries.

Related papers

• On Pre-training of Multimodal Language Models Customized for Chart Understanding [83.99377088129282]
  This paper explores the training processes necessary to improve MLLMs' comprehension of charts. We introduce CHOPINLLM, an MLLM tailored for in-depth chart comprehension.
  arXiv  Detail & Related papers  (2024-07-19T17:58:36Z)
• ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild [28.643565008567172]
  We introduce ChartGemma, a novel chart understanding and reasoning model developed over PaliGemma. Rather than relying on underlying data tables, ChartGemma is trained on instruction-tuning data generated directly from chart images. Our simple approach achieves state-of-the-art results across $5$ benchmarks spanning chart summarization, question answering, and fact-checking.
  arXiv  Detail & Related papers  (2024-07-04T22:16:40Z)
• When Graph Data Meets Multimodal: A New Paradigm for Graph Understanding and Reasoning [54.84870836443311]
  The paper presents a new paradigm for understanding and reasoning about graph data by integrating image encoding and multimodal technologies. This approach enables the comprehension of graph data through an instruction-response format, utilizing GPT-4V's advanced capabilities. The study evaluates this paradigm on various graph types, highlighting the model's strengths and weaknesses, particularly in Chinese OCR performance and complex reasoning tasks.
  arXiv  Detail & Related papers  (2023-12-16T08:14:11Z)
• VisText: A Benchmark for Semantically Rich Chart Captioning [12.117737635879037]
  VisText is a dataset of 12,441 pairs of charts and captions that describe the charts' construction. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models.
  arXiv  Detail & Related papers  (2023-06-28T15:16:24Z)
• ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries [0.26097841018267615]
  Automatic chart to text summarization is an effective tool for the visually impaired people. In this paper, we propose ChartSumm: a large-scale benchmark dataset consisting of a total of 84,363 charts.
  arXiv  Detail & Related papers  (2023-04-26T15:25:24Z)
• ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules [89.75395046894809]
  We present ChartReader, a unified framework that seamlessly integrates chart derendering and comprehension tasks. Our approach includes a transformer-based chart component detection module and an extended pre-trained vision-language model for chart-to-X tasks. Our proposed framework can significantly reduce the manual effort involved in chart analysis, providing a step towards a universal chart understanding model.
  arXiv  Detail & Related papers  (2023-04-05T00:25:27Z)
• Chart-to-Text: A Large-Scale Benchmark for Chart Summarization [9.647079534077472]
  We present Chart-to-text, a large-scale benchmark with two datasets and a total of 44,096 charts. We explain the dataset construction process and analyze the datasets.
  arXiv  Detail & Related papers  (2022-03-12T17:01:38Z)
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph [53.70520466556453]
  We propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow. In addition, a progressive learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph.
  arXiv  Detail & Related papers  (2021-05-06T12:20:41Z)
• Promoting Graph Awareness in Linearized Graph-to-Text Generation [72.83863719868364]
  We study the ability of linearized models to encode local graph structures. Our findings motivate solutions to enrich the quality of models' implicit graph encodings. We find that these denoising scaffolds lead to substantial improvements in downstream generation in low-resource settings.
  arXiv  Detail & Related papers  (2020-12-31T18:17:57Z)
• Scene Graph Modification Based on Natural Language Commands [90.0662899539489]
  Structured representations like graphs and parse trees play a crucial role in many Natural Language Processing systems. In this paper, we explore the novel problem of graph modification, where the systems need to learn how to update an existing graph given a new user's command.
  arXiv  Detail & Related papers  (2020-10-06T10:01:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.