FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets

• URL: http://arxiv.org/abs/2603.07316v1
• Date: Sat, 07 Mar 2026 19:27:34 GMT
• Title: FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets
• Abstract summary: FinSheet-Bench is a benchmark of synthetic financial portfolio data modeled on real private equity fund structures.<n>No standalone model achieves error rates low enough for unsupervised use in professional finance applications.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Large Language Models (LLMs) can accelerate text-heavy tasks in alternative investment due diligence, a gap remains in their ability to accurately extract and reason over structured tabular data from complex financial spreadsheets. Progress is held back by the lack of real industry fund portfolio datasets for benchmarking, as private equity data rooms are confidential. To address this, we introduce FinSheet-Bench, a benchmark of synthetic financial portfolio data modeled on real private equity fund structures, designed to evaluate LLM performance on text-serialized spreadsheet question answering and numeric reasoning tasks. Our evaluation of ten model configurations from OpenAI, Google, and Anthropic on financial spreadsheets, including complex layouts, fund dividers, and multi-line column names, reveals that no standalone model achieves error rates low enough for unsupervised use in professional finance applications. The best-performing model, Gemini 3.1 Pro, achieves 82.4% accuracy across twenty-four evaluation files of varying complexity and structural layout (approximately 1 error per 6 questions), followed by GPT-5.2 with reasoning at 80.4%, Claude Opus 4.6 with thinking at 80.2%, and Gemini 3 Pro at 80.2%. Performance degrades substantially on larger, more complex spreadsheets: the largest spreadsheet (152 companies, 8 funds) yields an average accuracy of just 48.6% across all models, compared to 86.2% on the easiest evaluation file. These difficulty patterns are consistent across all ten models, indicating that they reflect LLM limitations rather than idiosyncratic model weaknesses. Reliable financial spreadsheet extraction will likely require architectural approaches that separate document understanding from deterministic computation.

Related papers

• V-FiLLM: Verified Financial LLM Reasoning Benchmark [0.37189423451031356]
  We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from computation trees grounded in real tables.<n>V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size.<n>We show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems.
  arXiv  Detail & Related papers  (2026-08-11T15:18:47Z)
• SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows [14.11925465537949]
  We introduce textscSpreadsheetBench 2, a taxonomy-level benchmark for spreadsheet agents.<n>The benchmark is constructed from authentic business data, including financial reports and corporate filings.<n>Results show that current systems remain far from reliable on real-world use.
  arXiv  Detail & Related papers  (2026-06-29T08:33:52Z)
• When Tables Go Crazy: Evaluating Multimodal Models on French Financial Documents [3.4992819560032267]
  Vision-language models (VLMs) perform well on many document understanding tasks, yet their reliability in specialized, non-English domains remains underexplored.<n>We introduce Multimodal Finance Eval, the first multimodal benchmark for evaluating French financial document understanding.<n>The dataset contains 1,204 expert-validated questions spanning text extraction, table comprehension, chart interpretation, and multi-turn conversational reasoning.
  arXiv  Detail & Related papers  (2026-02-11T00:04:56Z)
• XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning [28.967959142733903]
  We introduce XFinBench, a novel benchmark to evaluate large language models' ability in solving financial problems.<n>O1 is the best-performing text-only model with an overall accuracy of 67.3%, but still lags significantly behind human experts with 12.5%.<n>We construct a knowledge bank with 3,032 finance terms for knowledge augmentation analysis, and find that relevant knowledge only brings consistent accuracy improvements to small open-source model.
  arXiv  Detail & Related papers  (2025-08-20T15:23:35Z)
• Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports [0.0]
  We propose a fine-tuned vision-language model (VLM) based on Qwen2.5-VL-7B.<n>Our approach includes a curated dataset of 2,152 image-text pairs with augmentations and a supervised fine-tuning strategy using LoRA.<n>Our model achieves a 92.20% overall accuracy on the criteria-based assessment and a 96.53% markdown TEDS score.
  arXiv  Detail & Related papers  (2025-08-04T04:54:00Z)
• FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information [47.37027539828975]
  FinTagging is the first comprehensive benchmark for structure-aware and full-scope.<n>tagging.<n> FinNI for numeric identification extracts numerical entities and their types from.<n>financial reports.<n>FinCL for concept linking maps each extracted entity to the corresponding concept in the full U.S. taxonomy.
  arXiv  Detail & Related papers  (2025-05-27T02:55:53Z)
• BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs [7.9458352414205295]
  Large language models excel in general tasks, yet assessing their reliability in logic-heavy, precision-critical domains like finance, law, and healthcare remains challenging.<n>We introduce BizFinBench, the first benchmark specifically designed to evaluate LLMs in real-world financial applications.<n> BizFinBench consists of 6,781 well-annotated queries in Chinese, spanning five dimensions: numerical calculation, reasoning, information extraction, prediction recognition, and knowledge-based question answering.
  arXiv  Detail & Related papers  (2025-05-26T03:23:02Z)
• Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications [88.96861155804935]
  We introduce textitOpen-FinLLMs, the first open-source multimodal financial LLMs.<n>FinLLaMA is pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs.<n>We evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings.
  arXiv  Detail & Related papers  (2024-08-20T16:15:28Z)
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs [62.84082370758761]
  CharXiv is a comprehensive evaluation suite involving 2,323 charts from arXiv papers. To ensure quality, all charts and questions are handpicked, curated, and verified by human experts. Results reveal a substantial, previously underestimated gap between the reasoning skills of the strongest proprietary model.
  arXiv  Detail & Related papers  (2024-06-26T17:50:11Z)
• Advancing Anomaly Detection: Non-Semantic Financial Data Encoding with LLMs [49.57641083688934]
  We introduce a novel approach to anomaly detection in financial data using Large Language Models (LLMs) embeddings. Our experiments demonstrate that LLMs contribute valuable information to anomaly detection as our models outperform the baselines.
  arXiv  Detail & Related papers  (2024-06-05T20:19:09Z)
• FinBen: A Holistic Financial Benchmark for Large Language Models [75.09474986283394]
  FinBen is the first extensive open-source evaluation benchmark, including 36 datasets spanning 24 financial tasks. FinBen offers several key innovations: a broader range of tasks and datasets, the first evaluation of stock trading, novel agent and Retrieval-Augmented Generation (RAG) evaluation, and three novel open-source evaluation datasets for text summarization, question answering, and stock trading.
  arXiv  Detail & Related papers  (2024-02-20T02:16:16Z)
• PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance [63.51545277822702]
  PIXIU is a comprehensive framework including the first financial large language model (LLMs) based on fine-tuning LLaMA with instruction data. We propose FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. We conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks.
  arXiv  Detail & Related papers  (2023-06-08T14:20:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.