DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems

• URL: http://arxiv.org/abs/2407.10701v1
• Date: Mon, 15 Jul 2024 13:17:42 GMT
• Title: DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems
• Authors: Anni Zou, Wenhao Yu, Hongming Zhang, Kaixin Ma, Deng Cai, Zhuosheng Zhang, Hai Zhao, Dong Yu,
• Abstract summary: We introduce DocBench, a benchmark to evaluate large language model (LLM)-based document reading systems. Our benchmark involves the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions.
• Score: 99.17123445211115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, there has been a growing interest among large language model (LLM) developers in LLM-based document reading systems, which enable users to upload their own documents and pose questions related to the document contents, going beyond simple reading comprehension tasks. Consequently, these systems have been carefully designed to tackle challenges such as file parsing, metadata extraction, multi-modal information understanding and long-context reading. However, no current benchmark exists to evaluate their performance in such scenarios, where a raw file and questions are provided as input, and a corresponding response is expected as output. In this paper, we introduce DocBench, a new benchmark designed to evaluate LLM-based document reading systems. Our benchmark involves a meticulously crafted process, including the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions. We evaluate both proprietary LLM-based systems accessible via web interfaces or APIs, and a parse-then-read pipeline employing open-source LLMs. Our evaluations reveal noticeable gaps between existing LLM-based document reading systems and human performance, underscoring the challenges of developing proficient systems. To summarize, DocBench aims to establish a standardized benchmark for evaluating LLM-based document reading systems under diverse real-world scenarios, thereby guiding future advancements in this research area.

Related papers

• Zero-Shot Dense Retrieval with Embeddings from Relevance Feedback [17.986392250269606]
  We introduce Real Document Embeddings from Relevance Feedback (ReDE-RF) Inspired by relevance feedback, ReDE-RF proposes to re-frame hypothetical document generation as a relevance estimation task. Our experiments show that ReDE-RF consistently surpasses state-of-the-art zero-shot dense retrieval methods.
  arXiv  Detail & Related papers  (2024-10-28T17:40:40Z)
• RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content [13.187520657952263]
  Large Language Models (LLMs) are trained on vast amounts of data, most of which is automatically scraped from the internet. evaluating models on test splits that might have leaked into the training set is prone to misleading conclusions. We introduce a new test dataset named RepLiQA, suited for question-answering and topic retrieval tasks.
  arXiv  Detail & Related papers  (2024-06-17T17:52:54Z)
• RepoQA: Evaluating Long Context Code Understanding [12.329233433333416]
  RepoQA is a benchmark to evaluate Large Language Models (LLMs) on long-context code understanding. RepoQA includes 500 code search tasks gathered from 50 popular repositories across 5 modern programming languages.
  arXiv  Detail & Related papers  (2024-06-10T05:15:30Z)
• RepEval: Effective Text Evaluation with LLM Representation [55.26340302485898]
  RepEval is a metric that leverages the projection of Large Language Models (LLMs) representations for evaluation. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
  arXiv  Detail & Related papers  (2024-04-30T13:50:55Z)
• Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks [76.43527940649939]
  We introduce Ada-LEval, a benchmark for evaluating the long-context understanding of large language models (LLMs) Ada-LEval includes two challenging subsets, TSort and BestAnswer, which enable a more reliable evaluation of LLMs' long context capabilities. We evaluate 4 state-of-the-art closed-source API models and 6 open-source models with Ada-LEval.
  arXiv  Detail & Related papers  (2024-04-09T17:30:48Z)
• Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answering [40.2758450304531]
  Open-domain question answering (ODQA) has emerged as a pivotal research spotlight in information systems. We propose a framework that formulates the ODQA process into three basic steps: query expansion, document selection, and answer generation. We introduce a novel prompt optimization algorithm to refine role-playing prompts and steer LLMs to produce higher-quality evidence and answers.
  arXiv  Detail & Related papers  (2024-03-08T11:09:13Z)
• UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models [73.73303148524398]
  Large language models (LLMs) may generate text that lacks consistency with human knowledge, leading to factual inaccuracies or textithallucination. We propose textttUFO, an LLM-based unified and flexible evaluation framework to verify facts against plug-and-play fact sources.
  arXiv  Detail & Related papers  (2024-02-22T16:45:32Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization [132.25202059478065]
  We benchmark large language models (LLMs) on instruction controllable text summarization. Our study reveals that instruction controllable text summarization remains a challenging task for LLMs.
  arXiv  Detail & Related papers  (2023-11-15T18:25:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.