Deep Learning based Visually Rich Document Content Understanding: A Survey

• URL: http://arxiv.org/abs/2408.01287v1
• Date: Fri, 2 Aug 2024 14:19:34 GMT
• Title: Deep Learning based Visually Rich Document Content Understanding: A Survey
• Authors: Yihao Ding, Jean Lee, Soyeon Caren Han,
• Abstract summary: Visually Rich Documents (VRDs) are essential in academia, finance, medical fields, and marketing. Deep learning has revolutionized this process, introducing models that leverage multimodal information vision, text, and layout. These models have achieved state-of-the-art performance across various downstream tasks.
• Score: 8.788354139674789
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visually Rich Documents (VRDs) are essential in academia, finance, medical fields, and marketing due to their multimodal information content. Traditional methods for extracting information from VRDs depend on expert knowledge and manual labor, making them costly and inefficient. The advent of deep learning has revolutionized this process, introducing models that leverage multimodal information vision, text, and layout along with pretraining tasks to develop comprehensive document representations. These models have achieved state-of-the-art performance across various downstream tasks, significantly enhancing the efficiency and accuracy of information extraction from VRDs. In response to the growing demands and rapid developments in Visually Rich Document Understanding (VRDU), this paper provides a comprehensive review of deep learning-based VRDU frameworks. We systematically survey and analyze existing methods and benchmark datasets, categorizing them based on adopted strategies and downstream tasks. Furthermore, we compare different techniques used in VRDU models, focusing on feature representation and fusion, model architecture, and pretraining methods, while highlighting their strengths, limitations, and appropriate scenarios. Finally, we identify emerging trends and challenges in VRDU, offering insights into future research directions and practical applications. This survey aims to provide a thorough understanding of VRDU advancements, benefiting both academic and industrial sectors.

Related papers

• Deploying Large Language Models With Retrieval Augmented Generation [0.21485350418225244]
  Retrieval Augmented Generation has emerged as a key approach for integrating knowledge from data sources outside of the large language model's training set. We present insights from the development and field-testing of a pilot project that integrates LLMs with RAG for information retrieval.
  arXiv  Detail & Related papers  (2024-11-07T22:11:51Z)
• Building and better understanding vision-language models: insights and future directions [8.230565679484128]
  This paper provides a comprehensive overview of the current state-of-the-art approaches to vision-language models. We then walk through the practical steps to build Idefics3-8B, a powerful VLM that significantly outperforms its predecessor Idefics2-8B. We release the model along with the datasets created for its training.
  arXiv  Detail & Related papers  (2024-08-22T17:47:24Z)
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models [71.25225058845324]
  Large Language Models (LLMs) have demonstrated revolutionary abilities in language understanding and generation. Retrieval-Augmented Generation (RAG) can offer reliable and up-to-date external knowledge. RA-LLMs have emerged to harness external and authoritative knowledge bases, rather than relying on the model's internal knowledge.
  arXiv  Detail & Related papers  (2024-05-10T02:48:45Z)
• Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions [11.786387517781328]
  Vision-Language Models (VLMs) are advanced models that can tackle more intricate tasks such as image captioning and visual question answering. Our classification organizes VLMs into three distinct categories: models dedicated to vision-language understanding, models that process multimodal inputs to generate unimodal (textual) outputs and models that both accept and produce multimodal inputs and outputs. We meticulously dissect each model, offering an extensive analysis of its foundational architecture, training data sources, as well as its strengths and limitations wherever possible.
  arXiv  Detail & Related papers  (2024-02-20T18:57:34Z)
• Generative Multi-Modal Knowledge Retrieval with Large Language Models [75.70313858231833]
  We propose an innovative end-to-end generative framework for multi-modal knowledge retrieval. Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases. We demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.
  arXiv  Detail & Related papers  (2024-01-16T08:44:29Z)
• Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models [33.50873478562128]
  Large Language Models (LLMs) bring forth challenges in the high consumption of computational, memory, energy, and financial resources. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs.
  arXiv  Detail & Related papers  (2024-01-01T01:12:42Z)
• A Study on the Implementation of Generative AI Services Using an Enterprise Data-Based LLM Application Architecture [0.0]
  This study presents a method for implementing generative AI services by utilizing the Large Language Models (LLM) application architecture. The research delves into strategies for mitigating the issue of inadequate data, offering tailored solutions. A significant contribution of this work is the development of a Retrieval-Augmented Generation (RAG) model.
  arXiv  Detail & Related papers  (2023-09-03T07:03:17Z)
• A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning [53.35317176453194]
  Data augmentation (DA) has become a widely used technique in visual RL for acquiring sample-efficient and generalizable policies. We present a principled taxonomy of the existing augmentation techniques used in visual RL and conduct an in-depth discussion on how to better leverage augmented data. As the first comprehensive survey of DA in visual RL, this work is expected to offer valuable guidance to this emerging field.
  arXiv  Detail & Related papers  (2022-10-10T11:01:57Z)
• Retrieval-Enhanced Machine Learning [110.5237983180089]
  We describe a generic retrieval-enhanced machine learning framework, which includes a number of existing models as special cases. REML challenges information retrieval conventions, presenting opportunities for novel advances in core areas, including optimization. REML research agenda lays a foundation for a new style of information access research and paves a path towards advancing machine learning and artificial intelligence.
  arXiv  Detail & Related papers  (2022-05-02T21:42:45Z)
• Self-Supervised Representation Learning: Introduction, Advances and Challenges [125.38214493654534]
  Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data.
  arXiv  Detail & Related papers  (2021-10-18T13:51:22Z)
• Video Super Resolution Based on Deep Learning: A Comprehensive Survey [87.30395002197344]
  We comprehensively investigate 33 state-of-the-art video super-resolution (VSR) methods based on deep learning. We propose a taxonomy and classify the methods into six sub-categories according to the ways of utilizing inter-frame information. We summarize and compare the performance of the representative VSR method on some benchmark datasets.
  arXiv  Detail & Related papers  (2020-07-25T13:39:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.