LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extraction

• URL: http://arxiv.org/abs/2606.13100v1
• Date: Thu, 11 Jun 2026 09:28:43 GMT
• Title: LEDGER: A Long-Context Benchmark of Corporate Annual Reports for Grounded Financial Retrieval and Extraction
• Abstract summary: LEDGER is a corpus of 4,999 digitized corporate annual reports.<n>Each report is labeled with 31 consolidated financial-context-relevant reports.<n>We derive three evaluation benchmarks spanning the difficulty spectrum.
• Score: 0.18569884696859787
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Finance reporting is a natural proving ground for large language models, and the very-long-context capabilities of recent models across all sizes make rigorous evaluation in this domain an increasingly pressing need. Yet most public financial resources reduce the task to plain-text SEC 10-K filings paired with a handful of question-answer items. We release LEDGER (Long-context Evaluation of Documents for Grounded Extraction and Retrieval), a corpus of 4,999 digitized corporate annual reports - full documents with figures, tables, and narrative, not just regulatory filings. Each report is labeled with 31 consolidated financial KPIs to be extracted and linked to the market's reaction at the earnings date. From this data we derive three evaluation benchmarks spanning the difficulty spectrum: a pure page-level KPI retrieval task with TREC-style relevance judgments over 118,048 questions in natural language, a conversational "needle-in-a-haystack" single-value lookup, and a full KPI extraction task, both from long, numerically dense reports. We additionally provide human OCR-quality annotations with inter-annotator agreement and the complete extraction, validation, and scoring toolchain. We further demonstrate the dataset's research utility with a case study linking CEO-letter rhetoric to post-publication market impact.

Related papers

• The Stanford EDGAR Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data [1.6404022072626985]
  Stanford EDGAR Filings dataset is a reconstruction of SEC filings into layout-faithful MultiMarkdown.<n>SEFD makes audited financial statements, risk disclosures, ownership reports, accounting notes, and market-moving event filings usable as long-context pretraining data.<n>We release SEFD-v1, a 152B-token initial public snapshot, and provide corpus-level analyses of a larger 18.5M-filing archive estimated at 550B tokens.
  arXiv  Detail & Related papers  (2026-06-16T17:22:34Z)
• Enhancing Business Analytics through Hybrid Summarization of Financial Reports [0.152292571922932]
  Financial reports and earnings communications contain large volumes of structured and semi structured information.<n>We present a hybrid summarization framework that combines extractive and abstractive techniques to produce concise and factually reliable summaries.<n>These findings support the development of practical summarization systems for distilling lengthy financial texts into usable business insights.
  arXiv  Detail & Related papers  (2025-12-28T16:25:12Z)
• How Far Are We from Genuinely Useful Deep Research Agents? [48.596990593729]
  Deep Research Agents (DRAs) aim to automatically produce analyst-level reports through iterative information retrieval and synthesis.<n>Current benchmarks for report synthesis suffer from task complexity and subjective metrics.<n>We present Fine-grained DEepResearch bench (FINDER), an enhanced benchmark consisting of 100 human-curated research tasks.
  arXiv  Detail & Related papers  (2025-12-01T17:58:59Z)
• Evaluating Large Language Models for Stance Detection on Financial Targets from SEC Filing Reports and Earnings Call Transcripts [45.13099538394587]
  We introduce a sentence-level corpus for stance detection focused on three core financial metrics: debt, earnings per share (EPS), and sales.<n>The sentences were extracted from Form 10-K annual reports and ECTs, and labeled for stance using the advanced ChatGPT-o3-pro model.<n>Using this corpus, we conduct a systematic evaluation of modern large language models (LLMs) using zero-shot, few-shot, and Chain-of-Thought (CoT) prompting strategies.
  arXiv  Detail & Related papers  (2025-10-27T16:03:20Z)
• FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering [57.18367828883773]
  FinAgentBench is a benchmark for evaluating agentic retrieval with multi-step reasoning in finance.<n>The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms.<n>We evaluate a suite of state-of-the-art models and demonstrate how targeted fine-tuning can significantly improve agentic retrieval performance.
  arXiv  Detail & Related papers  (2025-08-07T22:15:22Z)
• FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation [65.04104723843264]
  We present FinDER, an expert-generated dataset tailored for Retrieval-Augmented Generation (RAG) in finance.<n>FinDER focuses on annotating search-relevant evidence by domain experts, offering 5,703 query-evidence-answer triplets.<n>By challenging models to retrieve relevant information from large corpora, FinDER offers a more realistic benchmark for evaluating RAG systems.
  arXiv  Detail & Related papers  (2025-04-22T11:30:13Z)
• TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification [32.958143806547234]
  We introduce the Text pROVEnance (TROVE) challenge to trace each sentence of a target text back to specific source sentences.<n>To benchmark TROVE, we construct our dataset by leveraging three public datasets covering 11 diverse scenarios.<n>We evaluate 11 LLMs under direct prompting and retrieval-augmented paradigms.
  arXiv  Detail & Related papers  (2025-03-19T15:09:39Z)
• Evaluating Large Language Models on Financial Report Summarization: An Empirical Study [9.28042182186057]
  We conduct a comparative study on three state-of-the-art Large Language Models (LLMs) Our primary motivation is to explore how these models can be harnessed within finance, a field demanding precision, contextual relevance, and robustness against erroneous or misleading information. We introduce an innovative evaluation framework that integrates both quantitative metrics (e.g., precision, recall) and qualitative analyses (e.g., contextual fit, consistency) to provide a holistic view of each model's output quality.
  arXiv  Detail & Related papers  (2024-11-11T10:36:04Z)
• Instruction-Guided Bullet Point Summarization of Long Financial Earnings Call Transcripts [25.4439290862464]
  We study the problem of bullet point summarization of Earning Callum Transcripts (ECTs) using the recently released dataset. We leverage an unsupervised question-based extractive module followed by a parameter efficient instruction-tuned abstractive module to solve this task. Our proposed model FLAN-FinBPS achieves new state-of-the-art performances outperforming the strongest baseline with 14.88% average ROUGE score gain.
  arXiv  Detail & Related papers  (2024-05-03T16:33:16Z)
• FABLES: Evaluating faithfulness and content selection in book-length summarization [55.50680057160788]
  In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on book-length documents. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate.
  arXiv  Detail & Related papers  (2024-04-01T17:33:38Z)
• Enabling and Analyzing How to Efficiently Extract Information from Hybrid Long Documents with LLMs [48.87627426640621]
  This research focuses on harnessing the potential of Large Language Models to comprehend critical information from financial reports. We propose an Automated Financial Information Extraction framework that enhances LLMs' ability to comprehend and extract information from financial reports. Our framework is effectively validated on GPT-3.5 and GPT-4, yielding average accuracy increases of 53.94% and 33.77%, respectively.
  arXiv  Detail & Related papers  (2023-05-24T10:35:58Z)
• TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance [71.76018597965378]
  We build a new large-scale Question Answering dataset containing both Tabular And Textual data, named TAT-QA. We propose a novel QA model termed TAGOP, which is capable of reasoning over both tables and text.
  arXiv  Detail & Related papers  (2021-05-17T06:12:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.