Text Role Classification in Scientific Charts Using Multimodal Transformers

• URL: http://arxiv.org/abs/2402.14579v1
• Date: Thu, 8 Feb 2024 13:21:44 GMT
• Title: Text Role Classification in Scientific Charts Using Multimodal Transformers
• Authors: Hye Jin Kim, Nicolas Lell, Ansgar Scherp
• Abstract summary: Text role classification involves classifying the semantic role of textual elements within scientific charts. We propose to finetune two pretrained multimodal document layout analysis models, LayoutLMv3 and UDOP, on chart datasets. We investigate whether data augmentation and balancing methods help the performance of the models.
• Score: 4.605099852396135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text role classification involves classifying the semantic role of textual elements within scientific charts. For this task, we propose to finetune two pretrained multimodal document layout analysis models, LayoutLMv3 and UDOP, on chart datasets. The transformers utilize the three modalities of text, image, and layout as input. We further investigate whether data augmentation and balancing methods help the performance of the models. The models are evaluated on various chart datasets, and results show that LayoutLMv3 outperforms UDOP in all experiments. LayoutLMv3 achieves the highest F1-macro score of 82.87 on the ICPR22 test dataset, beating the best-performing model from the ICPR22 CHART-Infographics challenge. Moreover, the robustness of the models is tested on a synthetic noisy dataset ICPR22-N. Finally, the generalizability of the models is evaluated on three chart datasets, CHIME-R, DeGruyter, and EconBiz, for which we added labels for the text roles. Findings indicate that even in cases where there is limited training data, transformers can be used with the help of data augmentation and balancing methods. The source code and datasets are available on GitHub under https://github.com/hjkimk/text-role-classification

Related papers

• Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedback [37.275533538711436]
  We propose a hierarchical pipeline and a new dataset for chart generation. Our dataset, Text2Chart31, includes 31 unique plot types referring to the Matplotlib library. We introduce a reinforcement learning-based instruction tuning technique for chart generation tasks without requiring human feedback.
  arXiv  Detail & Related papers  (2024-10-05T07:25:56Z)
• ChartEye: A Deep Learning Framework for Chart Information Extraction [2.4936576553283287]
  In this study, we propose a deep learning-based framework that provides a solution for key steps in the chart information extraction pipeline. The proposed framework utilizes hierarchal vision transformers for the tasks of chart-type and text-role classification, while YOLOv7 for text detection. Our proposed framework achieves excellent performance at every stage with F1-scores of 0.97 for chart-type classification, 0.91 for text-role classification, and a mean Average Precision of 0.95 for text detection.
  arXiv  Detail & Related papers  (2024-08-28T20:22:39Z)
• Table Transformers for Imputing Textual Attributes [15.823533688884105]
  We propose a novel end-to-end approach called Table Transformers for Imputing Textual Attributes (TTITA) Our approach shows competitive performance outperforming baseline models such as recurrent neural networks and Llama2. We incorporate multi-task learning to simultaneously impute for heterogeneous columns, boosting the performance for text imputation.
  arXiv  Detail & Related papers  (2024-08-04T19:54:12Z)
• On Pre-training of Multimodal Language Models Customized for Chart Understanding [83.99377088129282]
  This paper explores the training processes necessary to improve MLLMs' comprehension of charts. We introduce CHOPINLLM, an MLLM tailored for in-depth chart comprehension.
  arXiv  Detail & Related papers  (2024-07-19T17:58:36Z)
• RanLayNet: A Dataset for Document Layout Detection used for Domain Adaptation and Generalization [36.973388673687815]
  RanLayNet is a synthetic document dataset enriched with automatically assigned labels. We show that a deep layout identification model trained on our dataset exhibits enhanced performance compared to a model trained solely on actual documents.
  arXiv  Detail & Related papers  (2024-04-15T07:50:15Z)
• ChartLlama: A Multimodal LLM for Chart Understanding and Generation [70.1393163657813]
  We create a high-quality instruction-tuning dataset leveraging GPT-4. Next, we introduce ChartLlama, a multi-modal large language model that we've trained using our created dataset.
  arXiv  Detail & Related papers  (2023-11-27T15:20:23Z)
• PixT3: Pixel-based Table-To-Text Generation [66.96636025277536]
  We present PixT3, a multimodal table-to-text model that overcomes the challenges of linearization and input size limitations. Experiments on the ToTTo and Logic2Text benchmarks show that PixT3 is competitive and superior to generators that operate solely on text.
  arXiv  Detail & Related papers  (2023-11-16T11:32:47Z)
• Zero-shot Composed Text-Image Retrieval [72.43790281036584]
  We consider the problem of composed image retrieval (CIR) It aims to train a model that can fuse multi-modal information, e.g., text and images, to accurately retrieve images that match the query, extending the user's expression ability.
  arXiv  Detail & Related papers  (2023-06-12T17:56:01Z)
• Transformer for Graphs: An Overview from Architecture Perspective [86.3545861392215]
  It's imperative to sort out the existing Transformer models for graphs and systematically investigate their effectiveness on various graph tasks. We first disassemble the existing models and conclude three typical ways to incorporate the graph information into the vanilla Transformer. Our experiments confirm the benefits of current graph-specific modules on Transformer and reveal their advantages on different kinds of graph tasks.
  arXiv  Detail & Related papers  (2022-02-17T06:02:06Z)
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields [83.43249184357053]
  We assemble 18 multimodal data tables that each contain some text fields. Our benchmark enables researchers to evaluate their own methods for supervised learning with numeric, categorical, and text features.
  arXiv  Detail & Related papers  (2021-11-04T09:29:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.