Back to the Future: A workbook time machine for spread sheet creation benchmarks

• URL: http://arxiv.org/abs/2608.07873v1
• Date: Sat, 08 Aug 2026 02:57:50 GMT
• Title: Back to the Future: A workbook time machine for spread sheet creation benchmarks
• Abstract summary: We introduce the workbook time machine, a pipeline that automatically creates benchmarks evaluating the ability of language models to create derived objects in spreadsheets.<n>applied to public workbook corpora, it produces wtmcorpus--a collection of (input workbook, output workbook, query) triples spanning four artifact types and varying complexity.<n>We evaluate existing spreadsheet manipulation agents and baselines on wtmbench across artifact types, step complexity, and instruction granularity.
• Score: 14.460401299531972
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce the workbook time machine, a pipeline that automatically creates benchmarks evaluating the ability of language models to create derived objects in spreadsheets (formulas, charts, pivot tables, and conditional formatting). Applied to public workbook corpora, it produces wtmcorpus--a collection of (input workbook, output workbook, query) triples spanning four artifact types and varying complexity. From this corpus we curate wtmbench, a 150-task evaluation benchmark with queries at three levels of specificity. We evaluate existing spreadsheet manipulation agents and baselines on wtmbench across artifact types, step complexity, and instruction granularity. Our evaluations show that query specificity, agent orchestration, and interface API used to control spreadsheets play a big role in LLM performance on Excel tasks.

Related papers

• SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows [14.11925465537949]
  We introduce textscSpreadsheetBench 2, a taxonomy-level benchmark for spreadsheet agents.<n>The benchmark is constructed from authentic business data, including financial reports and corporate filings.<n>Results show that current systems remain far from reliable on real-world use.
  arXiv  Detail & Related papers  (2026-06-29T08:33:52Z)
• SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks [8.306042091008957]
  Large language models (LLMs) are increasingly tasked with producing and manipulating structured artifacts.<n>We introduce SpreadsheetArena, a platform for evaluating models' performance on the task.<n>We observe that stylistic, structural, and functional features of preferred spreadsheets vary substantially across use cases.
  arXiv  Detail & Related papers  (2026-02-16T14:24:36Z)
• ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement [58.957050610762565]
  ShowTable is a pipeline that synergizes MLLMs with diffusion models via a progressive self-correcting process.<n> MLLM acts as the central orchestrator for reasoning the visual plan and judging visual errors.<n>We introduce TableVisBench, a new benchmark with 800 challenging instances across 5 evaluation dimensions.
  arXiv  Detail & Related papers  (2025-12-15T13:21:50Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation [34.8332394229927]
  SpreadsheetBench is designed to immerse current large language models (LLMs) in the actual workflow of spreadsheet users. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is built from 912 real questions gathered from online Excel forums. Our comprehensive evaluation of various LLMs under both single-round and multi-round inference settings reveals a substantial gap between the state-of-the-art (SOTA) models and human performance.
  arXiv  Detail & Related papers  (2024-06-21T09:06:45Z)
• TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools [51.576974932743596]
  Large Language Models (LLMs) often do not perform well on queries that require the aggregation of information across texts. TACT contains challenging instructions that demand stitching information scattered across one or more texts. We construct this dataset by leveraging an existing dataset of texts and their associated tables. We demonstrate that all contemporary LLMs perform poorly on this dataset, achieving an accuracy below 38%.
  arXiv  Detail & Related papers  (2024-06-05T20:32:56Z)
• TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios [51.66718740300016]
  TableLLM is a robust large language model (LLM) with 8 billion parameters.<n>TableLLM is purpose-built for proficiently handling data manipulation tasks.<n>We have released the model checkpoint, source code, benchmarks, and a web application for user interaction.
  arXiv  Detail & Related papers  (2024-03-28T11:21:12Z)
• NL2Formula: Generating Spreadsheet Formulas from Natural Language Queries [29.33149993368329]
  This paper introduces a novel benchmark task called NL2Formula. The aim is to generate executable formulas that are grounded on a spreadsheet table, given a Natural Language (NL) query as input. We construct a comprehensive dataset consisting of 70,799 paired NL queries and corresponding spreadsheet formulas, covering 21,670 tables and 37 types of formula functions.
  arXiv  Detail & Related papers  (2024-02-20T05:58:05Z)
• InstructExcel: A Benchmark for Natural Language Instruction in Excel [72.018640505825]
  This work investigates whether Large Language Models can generate code that solves Excel specific tasks provided via natural language user instructions. Our benchmark includes over 10k samples covering 170+ Excel operations across 2,000 publicly available Excel spreadsheets. We observe that (1) using GPT-4 over GPT-3.5, (2) providing more in-context examples, and (3) dynamic prompting can help improve performance on this benchmark.
  arXiv  Detail & Related papers  (2023-10-23T02:00:55Z)
• SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models [60.171444066848856]
  We propose a SheetCopilot agent that takes natural language task and control spreadsheet to fulfill the requirements. We curate a representative dataset containing 221 spreadsheet control tasks and establish a fully automated evaluation pipeline. Our SheetCopilot correctly completes 44.3% of tasks for a single generation, outperforming the strong code generation baseline by a wide margin.
  arXiv  Detail & Related papers  (2023-05-30T17:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.