GeoContrastNet: Contrastive Key-Value Edge Learning for Language-Agnostic Document Understanding

• URL: http://arxiv.org/abs/2405.03104v1
• Date: Mon, 6 May 2024 01:40:20 GMT
• Title: GeoContrastNet: Contrastive Key-Value Edge Learning for Language-Agnostic Document Understanding
• Authors: Nil Biescas, Carlos Boned, Josep Lladós, Sanket Biswas,
• Abstract summary: We present a language-agnostic framework to structured document understanding (DU) by integrating a contrastive learning objective with graph attention networks (GATs) We propose a novel methodology that combines geometric edge features with visual features within an overall two-staged GAT-based framework. Our results highlight the model's proficiency in identifying key-value relationships within the FUNSD dataset for forms and also discovering the spatial relationships in table-structured layouts for RVLCDIP business invoices.
• Score: 4.258365032282028
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This paper presents GeoContrastNet, a language-agnostic framework to structured document understanding (DU) by integrating a contrastive learning objective with graph attention networks (GATs), emphasizing the significant role of geometric features. We propose a novel methodology that combines geometric edge features with visual features within an overall two-staged GAT-based framework, demonstrating promising results in both link prediction and semantic entity recognition performance. Our findings reveal that combining both geometric and visual features could match the capabilities of large DU models that rely heavily on Optical Character Recognition (OCR) features in terms of performance accuracy and efficiency. This approach underscores the critical importance of relational layout information between the named text entities in a semi-structured layout of a page. Specifically, our results highlight the model's proficiency in identifying key-value relationships within the FUNSD dataset for forms and also discovering the spatial relationships in table-structured layouts for RVLCDIP business invoices. Our code and pretrained models will be accessible on our official GitHub.

Related papers

• ComAlign: Compositional Alignment in Vision-Language Models [2.3250871476216814]
  We introduce Compositional Alignment (ComAlign) to discover more exact correspondence of text and image components. Our methodology emphasizes that the compositional structure extracted from the text modality must also be retained in the image modality. We train a lightweight network lying on top of existing visual and language encoders using a small dataset.
  arXiv  Detail & Related papers  (2024-09-12T16:46:41Z)
• Hypergraph based Understanding for Document Semantic Entity Recognition [65.84258776834524]
  We build a novel hypergraph attention document semantic entity recognition framework, HGA, which uses hypergraph attention to focus on entity boundaries and entity categories at the same time. Our results on FUNSD, CORD, XFUNDIE show that our method can effectively improve the performance of semantic entity recognition tasks.
  arXiv  Detail & Related papers  (2024-07-09T14:35:49Z)
• MindBench: A Comprehensive Benchmark for Mind Map Structure Recognition and Analysis [35.31073435549237]
  We introduce the new benchmark named MindBench for document analysis. It includes meticulously constructed bilingual authentic or synthetic images, detailed annotations, evaluation metrics and baseline models. These tasks include full parsing, partial parsing, position-related parsing, structured Visual Question Answering (VQA), and position-related VQA.
  arXiv  Detail & Related papers  (2024-07-03T06:39:18Z)
• Unleashing the Potential of Text-attributed Graphs: Automatic Relation Decomposition via Large Language Models [31.443478448031886]
  RoSE (Relation-oriented Semantic Edge-decomposition) is a novel framework that decomposes the graph structure by analyzing raw text attributes. Our framework significantly enhances node classification performance across various datasets, with improvements of up to 16% on the Wisconsin dataset.
  arXiv  Detail & Related papers  (2024-05-28T20:54:47Z)
• Exploiting Contextual Target Attributes for Target Sentiment Classification [53.30511968323911]
  Existing PTLM-based models for TSC can be categorized into two groups: 1) fine-tuning-based models that adopt PTLM as the context encoder; 2) prompting-based models that transfer the classification task to the text/word generation task. We present a new perspective of leveraging PTLM for TSC: simultaneously leveraging the merits of both language modeling and explicit target-context interactions via contextual target attributes.
  arXiv  Detail & Related papers  (2023-12-21T11:45:28Z)
• Enhancing Visually-Rich Document Understanding via Layout Structure Modeling [91.07963806829237]
  We propose GraphLM, a novel document understanding model that injects layout knowledge into the model. We evaluate our model on various benchmarks, including FUNSD, XFUND and CORD, and achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2023-08-15T13:53:52Z)
• DocTr: Document Transformer for Structured Information Extraction in Documents [36.1145541816468]
  We present a new formulation for structured information extraction from visually rich documents. It aims to address the limitations of existing IOB tagging or graph-based formulations. We represent an entity as an anchor word and a bounding box, and represent entity linking as the association between anchor words.
  arXiv  Detail & Related papers  (2023-07-16T02:59:30Z)
• Deep Relational Metric Learning [84.95793654872399]
  This paper presents a deep relational metric learning framework for image clustering and retrieval. We learn an ensemble of features that characterizes an image from different aspects to model both interclass and intraclass distributions. Experiments on the widely-used CUB-200-2011, Cars196, and Stanford Online Products datasets demonstrate that our framework improves existing deep metric learning methods and achieves very competitive results.
  arXiv  Detail & Related papers  (2021-08-23T09:31:18Z)
• VSR: A Unified Framework for Document Layout Analysis combining Vision, Semantics and Relations [40.721146438291335]
  We propose a unified framework VSR for document layout analysis, combining vision, semantics and relations. On three popular benchmarks, VSR outperforms previous models by large margins.
  arXiv  Detail & Related papers  (2021-05-13T12:20:30Z)
• Language and Visual Entity Relationship Graph for Agent Navigation [54.059606864535304]
  Vision-and-Language Navigation (VLN) requires an agent to navigate in a real-world environment following natural language instructions. We propose a novel Language and Visual Entity Relationship Graph for modelling the inter-modal relationships between text and vision. Experiments show that by taking advantage of the relationships we are able to improve over state-of-the-art.
  arXiv  Detail & Related papers  (2020-10-19T08:25:55Z)
• Bidirectional Graph Reasoning Network for Panoptic Segmentation [126.06251745669107]
  We introduce a Bidirectional Graph Reasoning Network (BGRNet) to mine the intra-modular and intermodular relations within and between foreground things and background stuff classes. BGRNet first constructs image-specific graphs in both instance and semantic segmentation branches that enable flexible reasoning at the proposal level and class level.
  arXiv  Detail & Related papers  (2020-04-14T02:32:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.