ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents

• URL: http://arxiv.org/abs/2409.15004v1
• Date: Mon, 23 Sep 2024 13:28:06 GMT
• Title: ViBERTgrid BiLSTM-CRF: Multimodal Key Information Extraction from Unstructured Financial Documents
• Abstract summary: The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2 percentage points) on named entity recognition from unstructured documents in financial domain.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal key information extraction (KIE) models have been studied extensively on semi-structured documents. However, their investigation on unstructured documents is an emerging research topic. The paper presents an approach to adapt a multimodal transformer (i.e., ViBERTgrid previously explored on semi-structured documents) for unstructured financial documents, by incorporating a BiLSTM-CRF layer. The proposed ViBERTgrid BiLSTM-CRF model demonstrates a significant improvement in performance (up to 2 percentage points) on named entity recognition from unstructured documents in financial domain, while maintaining its KIE performance on semi-structured documents. As an additional contribution, we publicly released token-level annotations for the SROIE dataset in order to pave the way for its use in multimodal sequence labeling models.

Related papers

• Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomy [14.888842472004262]
  We construct the first Multi-level, Multi-domain, Multi-modal document classification Benchmark (MMM-Bench)<n>MMM-Bench includes (1) a deeply hierarchical taxonomy spanning five levels that capture the authentic organizational logic of business documentation; and (2) 5,990 real-world multi-modal documents meticulously curated from 12 commercial domains in Alibaba.
  arXiv  Detail & Related papers  (2026-05-11T13:28:27Z)
• MultiDocFusion: Hierarchical and Multimodal Chunking Pipeline for Enhanced RAG on Long Industrial Documents [30.113467423647865]
  RAG-based QA has emerged as a powerful method for processing long industrial documents.<n>We introduce MultiDocFusion, a multimodal chunking pipeline that integrates vision-based document parsing, OCR, text extraction and reconstruction.<n>Experiments demonstrate that MultiDocFusion improves retrieval precision by 8-15% and ANLS QA scores by 2-3% compared to baselines.
  arXiv  Detail & Related papers  (2026-04-14T06:40:22Z)
• Agentar-Fin-OCR [6.212573253529528]
  Agentar-Fin-OCR is a document parsing system tailored to financial-domain documents.<n>To address finance-specific challenges such as complex layouts, cross-page structural discontinuities, Agentar-Fin-OCR combines a Cross-page Contents Consolidation algorithm.<n>Experiments demonstrate that our model shows high performance on the table parsing metrics of OmniDocBench.
  arXiv  Detail & Related papers  (2026-03-11T17:59:42Z)
• ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images [19.490609860018804]
  We introduce ExStrucTiny, a new benchmark dataset for structured Information Extraction (IE) from document images.<n>Built through a novel pipeline combining manual and synthetic human-validated samples, ExStrucTiny covers more varied document types and extraction scenarios.<n>We analyze open and closed Vision Language Models on this benchmark, highlighting challenges such as adaptation, query under-specification, and schema adaptation.
  arXiv  Detail & Related papers  (2026-02-12T17:38:57Z)
• UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents [65.14244917622881]
  Recent Large Multimodal Models have shown promising potential for performing end-to-end KIE directly from document images.<n>We introduce UNIKIE-BENCH, a benchmark designed to rigorously evaluate the KIE capabilities of LMMs.<n>Experiments on 15 state-of-the-art LMMs reveal substantial performance degradation under diverse schema definitions, long-tail key fields, and complex layouts.
  arXiv  Detail & Related papers  (2026-02-03T12:04:56Z)
• MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns [80.05126590825121]
  MonkeyOCR v1.5 is a unified vision-language framework that enhances both layout understanding and content recognition.<n>To address complex table structures, we propose a visual consistency-based reinforcement learning scheme.<n>Two specialized modules, Image-Decoupled Table Parsing and Type-Guided Table Merging, are introduced to enable reliable parsing of tables.
  arXiv  Detail & Related papers  (2025-11-13T15:12:17Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• Benchmarking Multimodal Understanding and Complex Reasoning for ESG Tasks [56.350173737493215]
  Environmental, Social, and Governance (ESG) reports are essential for evaluating sustainability practices, ensuring regulatory compliance, and promoting financial transparency.<n>MMESGBench is a first-of-its-kind benchmark dataset to evaluate multimodal understanding and complex reasoning across structurally diverse and multi-source ESG documents.<n>MMESGBench comprises 933 validated QA pairs derived from 45 ESG documents, spanning across seven distinct document types and three major ESG source categories.
  arXiv  Detail & Related papers  (2025-07-25T03:58:07Z)
• Docopilot: Improving Multimodal Models for Document-Level Understanding [87.60020625241178]
  We present a high-quality document-level dataset, Doc-750K, designed to support in-depth understanding of multimodal documents.<n>This dataset includes diverse document structures, extensive cross-page dependencies, and real question-answer pairs derived from the original documents.<n>Building on the dataset, we develop a native multimodal model, Docopilot, which can accurately handle document-level dependencies without relying on RAG.
  arXiv  Detail & Related papers  (2025-07-19T16:03:34Z)
• DREAM: Document Reconstruction via End-to-end Autoregressive Model [53.51754520966657]
  We present an innovative autoregressive model specifically designed for document reconstruction, referred to as Document Reconstruction via End-to-end Autoregressive Model (DREAM)<n>We establish a standardized definition of the document reconstruction task, and introduce a novel Document Similarity Metric (DSM) and DocRec1K dataset for assessing the performance of the task.
  arXiv  Detail & Related papers  (2025-07-08T09:24:07Z)
• Unsupervised Document and Template Clustering using Multimodal Embeddings [0.0]
  This paper investigates a novel approach to unsupervised document clustering by leveraging multimodal embeddings as input.<n>Our method aims to achieve a finer-grained document understanding by grouping documents at the type level and distinguishing between different templates.<n>We evaluated the effectiveness of this approach using embeddings generated by several state-of-the-art pretrained multimodal models.
  arXiv  Detail & Related papers  (2025-06-13T14:07:44Z)
• VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation [100.06122876025063]
  This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings. We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG.
  arXiv  Detail & Related papers  (2024-12-14T06:24:55Z)
• Multi-Field Adaptive Retrieval [39.38972160512916]
  We introduce Multi-Field Adaptive Retrieval (MFAR), a flexible framework that accommodates any number of document indices on structured data. Our framework consists of two main steps: (1) the decomposition of an existing document into fields, each indexed independently through dense and lexical methods, and (2) learning a model which adaptively predicts the importance of a field by conditioning on the document query. We find that our approach allows for the optimized use of dense versus lexical representations across field types, significantly improves in document ranking over a number of existing retrievers, and achieves state-of-the-art performance for multi-field structured
  arXiv  Detail & Related papers  (2024-10-26T03:07:22Z)
• SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding [55.48936731641802]
  We present the SRFUND, a hierarchically structured multi-task form understanding benchmark. SRFUND provides refined annotations on top of the original FUNSD and XFUND datasets. The dataset includes eight languages including English, Chinese, Japanese, German, French, Spanish, Italian, and Portuguese.
  arXiv  Detail & Related papers  (2024-06-13T02:35:55Z)
• LLM Based Multi-Agent Generation of Semi-structured Documents from Semantic Templates in the Public Administration Domain [2.3999111269325266]
  Large Language Models (LLMs) have enabled the creation of customized text output satisfying user requests. We propose a novel approach that combines the LLMs with prompt engineering and multi-agent systems for generating new documents compliant with a desired structure.
  arXiv  Detail & Related papers  (2024-02-21T13:54:53Z)
• Document Structure in Long Document Transformers [64.76981299465885]
  Long documents often exhibit structure with hierarchically organized elements of different functions, such as section headers and paragraphs. Despite the omnipresence of document structure, its role in natural language processing (NLP) remains opaque. Do long-document Transformer models acquire an internal representation of document structure during pre-training? How can structural information be communicated to a model after pre-training, and how does it influence downstream performance?
  arXiv  Detail & Related papers  (2024-01-31T08:28:06Z)
• Benchmarking Diverse-Modal Entity Linking with Generative Models [78.93737257356784]
  We construct a benchmark for diverse-modal EL (DMEL) from existing EL datasets. To approach the DMEL task, we proposed a generative diverse-modal model (GDMM) following a multimodal-encoder-decoder paradigm. GDMM builds a stronger DMEL baseline, outperforming state-of-the-art task-specific EL models by 8.51 F1 score on average.
  arXiv  Detail & Related papers  (2023-05-27T02:38:46Z)
• Doc2SoarGraph: Discrete Reasoning over Visually-Rich Table-Text Documents via Semantic-Oriented Hierarchical Graphs [79.0426838808629]
  We propose TAT-DQA, i.e. to answer the question over a visually-rich table-text document. Specifically, we propose a novel Doc2SoarGraph framework with enhanced discrete reasoning capability. We conduct extensive experiments on TAT-DQA dataset, and the results show that our proposed framework outperforms the best baseline model by 17.73% and 16.91% in terms of Exact Match (EM) and F1 score respectively on the test set.
  arXiv  Detail & Related papers  (2023-05-03T07:30:32Z)
• Generating a Structured Summary of Numerous Academic Papers: Dataset and Method [20.90939310713561]
  We propose BigSurvey, the first large-scale dataset for generating comprehensive summaries of numerous academic papers on each topic. We collect target summaries from more than seven thousand survey papers and utilize their 430 thousand reference papers' abstracts as input documents. To organize the diverse content from dozens of input documents, we propose a summarization method named category-based alignment and sparse transformer (CAST)
  arXiv  Detail & Related papers  (2023-02-09T11:42:07Z)
• VRDU: A Benchmark for Visually-rich Document Understanding [22.040372755535767]
  We identify the desiderata for a more comprehensive benchmark and propose one we call Visually Rich Document Understanding (VRDU) VRDU contains two datasets that represent several challenges: rich schema including diverse data types as well as hierarchical entities, complex templates including tables and multi-column layouts, and diversity of different layouts (templates) within a single document type. We design few-shot and conventional experiment settings along with a carefully designed matching algorithm to evaluate extraction results.
  arXiv  Detail & Related papers  (2022-11-15T03:17:07Z)
• BERT-Flow-VAE: A Weakly-supervised Model for Multi-Label Text Classification [0.5156484100374058]
  We propose BERT-Flow-VAE (BFV), a Weakly-Supervised Multi-Label Text Classification model that reduces the need for full supervision. Experimental results on 6 multi-label datasets show that BFV can substantially outperform other baseline WSMLTC models in key metrics.
  arXiv  Detail & Related papers  (2022-10-27T07:18:56Z)
• Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval [79.37614949970013]
  We propose a new dense retrieval model which learns diverse document representations with deep query interactions. Our model encodes each document with a set of generated pseudo-queries to get query-informed, multi-view document representations.
  arXiv  Detail & Related papers  (2022-08-08T16:00:55Z)
• Data Augmentation for Abstractive Query-Focused Multi-Document Summarization [129.96147867496205]
  We present two QMDS training datasets, which we construct using two data augmentation methods. These two datasets have complementary properties, i.e., QMDSCNN has real summaries but queries are simulated, while QMDSIR has real queries but simulated summaries. We build end-to-end neural network models on the combined datasets that yield new state-of-the-art transfer results on DUC datasets.
  arXiv  Detail & Related papers  (2021-03-02T16:57:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.