GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling

• URL: http://arxiv.org/abs/2505.00063v2
• Date: Thu, 22 May 2025 05:16:26 GMT
• Title: GDI-Bench: A Benchmark for General Document Intelligence with Vision and Reasoning Decoupling
• Authors: Siqi Li, Yufan Shen, Xiangnan Chen, Jiayi Chen, Hengwei Ju, Haodong Duan, Song Mao, Hongbin Zhou, Bo Zhang, Bin Fu, Pinlong Cai, Licheng Wen, Botian Shi, Yong Liu, Xinyu Cai, Yu Qiao,
• Abstract summary: General Document Intelligence Benchmark features 2.3k images across 9 key scenarios and 19 document-specific tasks.<n>We evaluate various open-source and closed-source models on GDI-Bench, conducting decoupled analyses in the visual and reasoning domains.<n>Our model achieves state-of-the-art performance on previous benchmarks and the GDI-Bench.
• Score: 36.8157293625143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid advancement of multimodal large language models (MLLMs) has profoundly impacted the document domain, creating a wide array of application scenarios. This progress highlights the need for a comprehensive benchmark to evaluate these models' capabilities across various document-specific tasks. However, existing benchmarks often fail to locate specific model weaknesses or guide systematic improvements. To bridge this gap, we introduce a General Document Intelligence Benchmark (GDI-Bench), featuring 2.3k images across 9 key scenarios and 19 document-specific tasks. By decoupling visual complexity and reasoning complexity, the GDI-Bench structures graded tasks that allow performance assessment by difficulty, aiding in model weakness identification and optimization guidance. We evaluate various open-source and closed-source models on GDI-Bench, conducting decoupled analyses in the visual and reasoning domains, revealing their strengths and weaknesses. To address the diverse tasks and domains in the GDI-Bench, we propose a GDI-Model that mitigates catastrophic forgetting during the supervised fine-tuning (SFT) process through an intelligence-preserving training strategy, thereby reinforcing the inherent weaknesses of the base model. Our model achieves state-of-the-art performance on previous benchmarks and the GDI-Bench. Both our benchmark and models are or will be open-sourced on https://huggingface.co/GDIBench.

Related papers

• GLiDRE: Generalist Lightweight model for Document-level Relation Extraction [0.5130175508025212]
  We introduce GLiDRE, a new model for document-level relation extraction.<n>We benchmark GLiDRE against state-of-the-art models across various data settings on the Re-DocRED dataset.<n>Our results demonstrate that GLiDRE achieves state-of-the-art performance in few-shot scenarios.
  arXiv  Detail & Related papers  (2025-08-01T16:33:13Z)
• Generative Recommendation with Semantic IDs: A Practitioner's Handbook [34.25784373770595]
  Generative Recommendation (GR) has gained increasing attention for its promising performance compared to traditional models.<n>A key factor contributing to the success of GR is the semantic ID (SID), which converts continuous semantic representations into discrete ID sequences.<n>Our work introduces and open-sources a framework for Generative Recommendation with semantic ID, namely GRID, specifically designed for modularity.
  arXiv  Detail & Related papers  (2025-07-29T20:41:51Z)
• CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks [12.465309397733249]
  Large language models (LLMs) have been widely adopted across diverse software engineering domains.<n>These applications require understanding beyond surface-level code patterns.<n>Existing benchmarks primarily evaluate end-to-end outcomes, such as whether code is correctly repaired or generated.
  arXiv  Detail & Related papers  (2025-07-03T01:35:58Z)
• OpenGT: A Comprehensive Benchmark For Graph Transformers [13.214504021335749]
  Graph Transformers (GTs) have recently demonstrated remarkable performance across diverse domains.<n>This paper introduces OpenGT, a comprehensive benchmark for Graph Transformers.
  arXiv  Detail & Related papers  (2025-06-05T08:48:46Z)
• VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation [100.06122876025063]
  This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings.<n>We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG.
  arXiv  Detail & Related papers  (2024-12-14T06:24:55Z)
• DiscoveryBench: Towards Data-Driven Discovery with Large Language Models [50.36636396660163]
  We present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
  arXiv  Detail & Related papers  (2024-07-01T18:58:22Z)
• Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning [15.296263261737026]
  We introduce a Multi-Image MIRB Benchmark to evaluate visual language models' ability to compare, analyze, and reason across multiple images. Our benchmark encompasses four categories: perception, visual world knowledge, reasoning, and multi-hop reasoning. We demonstrate that while open-source VLMs were shown to approach the GPT-4V in single-image tasks, a significant gap remains in multi-image reasoning tasks.
  arXiv  Detail & Related papers  (2024-06-18T16:02:18Z)
• Global Benchmark Database [0.0]
  Global Benchmark Database (GBD) is a comprehensive suite of tools for provisioning and sustainably maintaining benchmark instances and their metadata. This paper introduces the data model of GBD as well as its interfaces and provides examples of how to interact with them.
  arXiv  Detail & Related papers  (2024-05-16T12:29:12Z)
• Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation [51.99752147380505]
  This paper presents a benchmark self-evolving framework to dynamically evaluate Large Language Models (LLMs) We utilize a multi-agent system to manipulate the context or question of original instances, reframing new evolving instances with high confidence. Our framework widens performance discrepancies both between different models and within the same model across various tasks.
  arXiv  Detail & Related papers  (2024-02-18T03:40:06Z)
• MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis [22.27724733876081]
  We present a Multi-Instance Generation (MIG) task, simultaneously generating multiple instances with diverse controls in one image. We introduce an innovative approach named Multi-Instance Generation Controller (MIGC) to address the challenges of the MIG task. To evaluate how well generation models perform on the MIG task, we provide a COCO-MIG benchmark along with an evaluation pipeline.
  arXiv  Detail & Related papers  (2024-02-08T04:52:36Z)
• CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks [20.390672895839757]
  Retrieval-augmented generation (RAG) has emerged as a popular solution to enhance factual accuracy. Traditional retrieval modules often rely on large document index and disconnect with generative tasks. We propose textbfCorpusLM, a unified language model that integrates generative retrieval, closed-book generation, and RAG.
  arXiv  Detail & Related papers  (2024-02-02T06:44:22Z)
• General Object Foundation Model for Images and Videos at Scale [99.2806103051613]
  We present GLEE, an object-level foundation model for locating and identifying objects in images and videos. GLEE accomplishes detection, segmentation, tracking, grounding, and identification of arbitrary objects in the open world scenario. We employ an image encoder, text encoder, and visual prompter to handle multi-modal inputs, enabling to simultaneously solve various object-centric downstream tasks.
  arXiv  Detail & Related papers  (2023-12-14T17:26:00Z)
• On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval [59.25292920967197]
  Few-shot document entity retrieval (VDER) is an important topic in industrial NLP applications. FewVEX is a new dataset to boost future research in the field of entity-level few-shot VDER. We present a task-aware meta-learning based framework, with a central focus on achieving effective task personalization.
  arXiv  Detail & Related papers  (2023-11-01T17:51:43Z)
• Striking Gold in Advertising: Standardization and Exploration of Ad Text Generation [5.3558730908641525]
  We propose a first benchmark dataset, CAMERA, to standardize the task of ATG. Our experiments show the current state and the remaining challenges. We also explore how existing metrics in ATG and an LLM-based evaluator align with human evaluations.
  arXiv  Detail & Related papers  (2023-09-21T12:51:24Z)
• Scalable Performance Analysis for Vision-Language Models [26.45624201546282]
  Joint vision-language models have shown great performance over a diverse set of tasks. Our paper introduces a more scalable solution that relies on already annotated benchmarks. We confirm previous findings that CLIP behaves like a bag of words model and performs better with nouns and verbs.
  arXiv  Detail & Related papers  (2023-05-30T06:40:08Z)
• Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction [67.54420015049732]
  Aspect Sentiment Triplet Extraction (ASTE) is a challenging task in sentiment analysis, aiming to provide fine-grained insights into human sentiments. Existing benchmarks are limited to two domains and do not evaluate model performance on unseen domains. We introduce a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings.
  arXiv  Detail & Related papers  (2023-05-23T18:01:49Z)
• KILT: a Benchmark for Knowledge Intensive Language Tasks [102.33046195554886]
  We present a benchmark for knowledge-intensive language tasks (KILT) All tasks in KILT are grounded in the same snapshot of Wikipedia. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline.
  arXiv  Detail & Related papers  (2020-09-04T15:32:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.