GraphRevisedIE: Multimodal Information Extraction with Graph-Revised Network

• URL: http://arxiv.org/abs/2410.01160v1
• Date: Wed, 2 Oct 2024 01:29:49 GMT
• Title: GraphRevisedIE: Multimodal Information Extraction with Graph-Revised Network
• Authors: Panfeng Cao, Jian Wu,
• Abstract summary: Key information extraction from visually rich documents (VRD) has been a challenging task in document intelligence. We propose a light-weight model named GraphIE that effectively embeds multimodal features such as textual, visual, and layout features from VRD. Extensive experiments on multiple real-world datasets show that GraphIEReviseds generalizes to documents of varied layouts and achieves comparable or better performance compared to previous KIE methods.
• Score: 3.9472311338123287
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Key information extraction (KIE) from visually rich documents (VRD) has been a challenging task in document intelligence because of not only the complicated and diverse layouts of VRD that make the model hard to generalize but also the lack of methods to exploit the multimodal features in VRD. In this paper, we propose a light-weight model named GraphRevisedIE that effectively embeds multimodal features such as textual, visual, and layout features from VRD and leverages graph revision and graph convolution to enrich the multimodal embedding with global context. Extensive experiments on multiple real-world datasets show that GraphRevisedIE generalizes to documents of varied layouts and achieves comparable or better performance compared to previous KIE methods. We also publish a business license dataset that contains both real-life and synthesized documents to facilitate research of document KIE.

Related papers

• Multimodal Graph Benchmark [36.75510196380185]
  Multimodal Graph Benchmark (MM-GRAPH) is first comprehensive multi-modal graph benchmark that incorporates both textual and visual information. MM-GRAPH consists of five graph learning datasets of various scales that are appropriate for different learning tasks. MM-GRAPH aims to foster research on multimodal graph learning and drive the development of more advanced and robust graph learning algorithms.
  arXiv  Detail & Related papers  (2024-06-24T05:14:09Z)
• Multi-Head RAG: Solving Multi-Aspect Problems with LLMs [13.638439488923671]
  Retrieval Augmented Generation (RAG) enhances the abilities of Large Language Models (LLMs) Existing RAG solutions do not focus on queries that may require fetching multiple documents with substantially different contents. This paper introduces Multi-Head RAG (MRAG), a novel scheme designed to address this gap with a simple yet powerful idea.
  arXiv  Detail & Related papers  (2024-06-07T16:59:38Z)
• All in One Framework for Multimodal Re-identification in the Wild [58.380708329455466]
  multimodal learning paradigm for ReID introduced, referred to as All-in-One (AIO) AIO harnesses a frozen pre-trained big model as an encoder, enabling effective multimodal retrieval without additional fine-tuning. Experiments on cross-modal and multimodal ReID reveal that AIO not only adeptly handles various modal data but also excels in challenging contexts.
  arXiv  Detail & Related papers  (2024-05-08T01:04:36Z)
• Enhancing Visually-Rich Document Understanding via Layout Structure Modeling [91.07963806829237]
  We propose GraphLM, a novel document understanding model that injects layout knowledge into the model. We evaluate our model on various benchmarks, including FUNSD, XFUND and CORD, and achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2023-08-15T13:53:52Z)
• Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling [96.75821232222201]
  Existing research on multimodal relation extraction (MRE) faces two co-existing challenges, internal-information over-utilization and external-information under-exploitation. We propose a novel framework that simultaneously implements the idea of internal-information screening and external-information exploiting.
  arXiv  Detail & Related papers  (2023-05-19T14:56:57Z)
• Document Understanding Dataset and Evaluation (DUDE) [29.78902147806488]
  Document Understanding dataset and evaluation (DUDE) seeks to remediate the halted research progress in understanding visually-rich documents (VRDs) We present a new dataset with novelties related to types of questions, answers, and document layouts based on multi-industry, multi-domain, and multi-page VRDs of various origins, and dates.
  arXiv  Detail & Related papers  (2023-05-15T08:54:32Z)
• Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion [112.27103169303184]
  Multimodal Knowledge Graphs (MKGs) organize visual-text factual knowledge. MKGformer can obtain SOTA performance on four datasets of multimodal link prediction, multimodal RE, and multimodal NER.
  arXiv  Detail & Related papers  (2022-05-04T23:40:04Z)
• One-shot Key Information Extraction from Document with Deep Partial Graph Matching [60.48651298832829]
  Key Information Extraction (KIE) from documents improves efficiency, productivity, and security in many industrial scenarios. Existing supervised learning methods for the KIE task need to feed a large number of labeled samples and learn separate models for different types of documents. We propose a deep end-to-end trainable network for one-shot KIE using partial graph matching.
  arXiv  Detail & Related papers  (2021-09-26T07:45:53Z)
• Multi-modal Summarization for Video-containing Documents [23.750585762568665]
  We propose a novel multi-modal summarization task to summarize from a document and its associated video. Comprehensive experiments show that the proposed model is beneficial for multi-modal summarization and superior to existing methods.
  arXiv  Detail & Related papers  (2020-09-17T02:13:14Z)
• Leveraging Graph to Improve Abstractive Multi-Document Summarization [50.62418656177642]
  We develop a neural abstractive multi-document summarization (MDS) model which can leverage well-known graph representations of documents. Our model utilizes graphs to encode documents in order to capture cross-document relations, which is crucial to summarizing long documents. Our model can also take advantage of graphs to guide the summary generation process, which is beneficial for generating coherent and concise summaries.
  arXiv  Detail & Related papers  (2020-05-20T13:39:47Z)
• PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks [5.210482046387142]
  Key Information Extraction from documents remains a challenge. We introduce PICK, a framework that is effective and robust in handling complex documents layout for KIE. Our method outperforms baselines methods by significant margins.
  arXiv  Detail & Related papers  (2020-04-16T05:20:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.