Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing

• URL: http://arxiv.org/abs/2603.06503v1
• Date: Fri, 06 Mar 2026 17:36:39 GMT
• Title: Beyond Rows to Reasoning: Agentic Retrieval for Multimodal Spreadsheet Understanding and Editing
• Abstract summary: We introduce Beyond Rows to Reasoning (BRTR), a framework for spreadsheet understanding that replaces single-pass retrieval with an iterative tool-calling loop.<n>BRTR achieves state-of-the-art performance across three frontier spreadsheet understanding benchmarks.
• Score: 0.7723674433972977
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in multimodal Retrieval-Augmented Generation (RAG) enable Large Language Models (LLMs) to analyze enterprise spreadsheet workbooks containing millions of cells, cross-sheet dependencies, and embedded visual artifacts. However, state-of-the-art approaches exclude critical context through single-pass retrieval, lose data resolution through compression, and exceed LLM context windows through naive full-context injection, preventing reliable multi-step reasoning over complex enterprise workbooks. We introduce Beyond Rows to Reasoning (BRTR), a multimodal agentic framework for spreadsheet understanding that replaces single-pass retrieval with an iterative tool-calling loop, supporting end-to-end Excel workflows from complex analysis to structured editing. Supported by over 200 hours of expert human evaluation, BRTR achieves state-of-the-art performance across three frontier spreadsheet understanding benchmarks, surpassing prior methods by 25 percentage points on FRTR-Bench, 7 points on SpreadsheetLLM, and 32 points on FINCH. We evaluate five multimodal embedding models, identifying NVIDIA NeMo Retriever 1B as the top performer for mixed tabular and visual data, and vary nine LLMs. Ablation experiments confirm that the planner, retrieval, and iterative reasoning each contribute substantially, and cost analysis shows GPT-5.2 achieves the best efficiency-accuracy trade-off. Throughout all evaluations, BRTR maintains full auditability through explicit tool-call traces.

Related papers

• SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows [14.11925465537949]
  We introduce textscSpreadsheetBench 2, a taxonomy-level benchmark for spreadsheet agents.<n>The benchmark is constructed from authentic business data, including financial reports and corporate filings.<n>Results show that current systems remain far from reliable on real-world use.
  arXiv  Detail & Related papers  (2026-06-29T08:33:52Z)
• Struct-Searcher: Agentic Structural Thinking Advances Multimodal Deep Information Seeking [57.25141826641528]
  Struct-Searcher is a structural agentic workflow grounded in belief theory.<n>It explicitly maintains an evolving multimodal structural graph throughout the reasoning process.
  arXiv  Detail & Related papers  (2026-06-05T06:25:11Z)
• MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning [74.07254720088926]
  We introduce MM-Doc-R1, a novel framework that employs an agentic, vision-aware workflow to address long document visual question answering.<n>We propose Similarity-based Policy Optimization (SPO), addressing baseline estimation bias in existing multi-turn reinforcement learning (RL) algorithms like GRPO.<n>Our experiments on the MMLongbench-Doc benchmark show that MM-Doc-R1 outperforms previous baselines by 10.4%.
  arXiv  Detail & Related papers  (2026-04-15T07:39:08Z)
• From Rows to Reasoning: A Retrieval-Augmented Multimodal Framework for Spreadsheet Understanding [0.7723674433972977]
  Large Language Models (LLMs) struggle to reason over large-scale enterprise spreadsheets containing thousands of numeric rows, multiple linked sheets, and embedded visual content such as charts and receipts.<n>We present From Rows to Reasoning (FRTR), an advanced, multimodal retrieval-augmented generation framework that decomposes Excel workbooks into granular row, column, and block embeddings.<n>We tested FRTR on six LLMs, achieving 74% answer accuracy on FRTR-Bench with Claude Sonnet 4.5, a substantial improvement over prior state-of-the-art approaches that reached only 24%.
  arXiv  Detail & Related papers  (2026-01-13T17:18:14Z)
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding [49.26132236798123]
  Vision Language Models (VLMs) have gradually become a primary approach in document understanding.<n>We propose SLEUTH, a multi agent framework that orchestrates a retriever and four collaborative agents in a coarse to fine process.<n>The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy.
  arXiv  Detail & Related papers  (2025-11-28T03:09:40Z)
• Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems [0.0]
  We present a comparative analysis of two retrieval approaches for multimodal RAG systems.<n>Direct multimodal embedding retrieval significantly outperforms LLM-summary-based approaches.
  arXiv  Detail & Related papers  (2025-11-20T18:56:49Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• MuaLLM: A Multimodal Large Language Model Agent for Circuit Design Assistance with Hybrid Contextual Retrieval-Augmented Generation [0.0]
  MuaLLM is an open-source multimodal Large Language Model (LLM) agent for circuit design assistance.<n>It integrates a hybrid Retrieval-Augmented Generation framework with an adaptive vector database of circuit design research papers.<n>It functions as a question-answering design assistant, capable of interpreting complex queries and providing reasoned responses.
  arXiv  Detail & Related papers  (2025-08-11T16:11:09Z)
• DABstep: Data Agent Benchmark for Multi-step Reasoning [2.6709582216950767]
  DABstep is a novel benchmark for evaluating AI agents on realistic multi-step data analysis tasks.<n>It comprises over 450 real-world challenges derived from a financial analytics platform.<n>Dabstep is released with a public leaderboard and toolkit to accelerate research in autonomous data analysis.
  arXiv  Detail & Related papers  (2025-06-30T10:49:21Z)
• IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis [60.32962597618861]
  IDA-Bench is a novel benchmark evaluating large language models in multi-round interactive scenarios.<n>Agent performance is judged by comparing its final numerical output to the human-derived baseline.<n>Even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on 50% of the tasks, highlighting limitations not evident in single-turn tests.
  arXiv  Detail & Related papers  (2025-05-23T09:37:52Z)
• Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts [56.7225771305861]
  This paper introduces Multi-Modal Retrieval-Augmented Generation (M$2$RAG), a benchmark designed to evaluate the effectiveness of Multi-modal Large Language Models.<n>The benchmark comprises four tasks: image captioning, multi-modal question answering, multi-modal fact verification, and image reranking.<n>To enhance the context utilization capabilities of MLLMs, we also introduce Multi-Modal Retrieval-Augmented Instruction Tuning (MM-RAIT)
  arXiv  Detail & Related papers  (2025-02-24T16:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.