FreshTab: Sourcing Fresh Data for Table-to-Text Generation Evaluation

• URL: http://arxiv.org/abs/2510.13598v1
• Date: Wed, 15 Oct 2025 14:31:44 GMT
• Title: FreshTab: Sourcing Fresh Data for Table-to-Text Generation Evaluation
• Authors: Kristýna Onderková, Ondřej Plátek, Zdeněk Kasner, Ondřej Dušek,
• Abstract summary: FreshTab is an on-the-fly table-to-text benchmark generation from Wikipedia.<n>We introduce FreshTab to combat the LLM data contamination problem and enable domain-sensitive evaluation.
• Score: 0.1749935196721634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Table-to-text generation (insight generation from tables) is a challenging task that requires precision in analyzing the data. In addition, the evaluation of existing benchmarks is affected by contamination of Large Language Model (LLM) training data as well as domain imbalance. We introduce FreshTab, an on-the-fly table-to-text benchmark generation from Wikipedia, to combat the LLM data contamination problem and enable domain-sensitive evaluation. While non-English table-to-text datasets are limited, FreshTab collects datasets in different languages on demand (we experiment with German, Russian and French in addition to English). We find that insights generated by LLMs from recent tables collected by our method appear clearly worse by automatic metrics, but this does not translate into LLM and human evaluations. Domain effects are visible in all evaluations, showing that a~domain-balanced benchmark is more challenging.

Related papers

• TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models [30.26407735827857]
  Reasoning with table-structured data poses significant challenges for large language models (LLMs)<n>We present a comprehensive table reasoning evolution benchmark, TReB, which measures both shallow table understanding abilities and deep table reasoning abilities.<n>We create an evaluation framework to robustly measure table reasoning capabilities with three distinct inference modes, TCoT, PoT and ICoT.
  arXiv  Detail & Related papers  (2025-06-23T09:02:04Z)
• TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering [41.12322828986314]
  Existing TableQA benchmarks are often limited by their focus on simple flat tables and suffer from data leakage.<n>We introduce TableEval, a new benchmark designed to evaluate LLMs on realistic TableQA tasks.<n>To minimize the risk of data leakage, we collect all data from recent real-world documents.
  arXiv  Detail & Related papers  (2025-06-04T13:39:01Z)
• Protecting multimodal large language models against misleading visualizations [94.71976205962527]
  We show that questionanswering (QA) accuracy on misleading visualizations drops on average to the level of the random baseline.<n>We introduce the first inference-time methods to improve QA performance on misleading visualizations, without compromising accuracy on non-misleading ones.<n>We find that two methods, table-based QA and redrawing the visualization, are effective, with improvements of up to 19.6 percentage points.
  arXiv  Detail & Related papers  (2025-02-27T20:22:34Z)
• TableRAG: Million-Token Table Understanding with Language Models [53.039560091592215]
  TableRAG is a Retrieval-Augmented Generation (RAG) framework specifically designed for LM-based table understanding.<n>TableRAG leverages query expansion combined with schema and cell retrieval to pinpoint crucial information before providing it to the LMs.<n>Our results demonstrate that TableRAG achieves the highest retrieval quality, leading to the new state-of-the-art performance on large-scale table understanding.
  arXiv  Detail & Related papers  (2024-10-07T04:15:02Z)
• Robustness of Large Language Models to Perturbations in Text [2.2734015467359217]
  Large language models (LLMs) have shown impressive performance, but can they handle the inevitable noise in real-world data?<n>This work tackles this critical question by investigating LLMs' resilience against morphological variations in text.<n>Our findings show that contrary to popular beliefs, generative LLMs are quiet robust to noisy perturbations in text.
  arXiv  Detail & Related papers  (2024-07-12T04:50:17Z)
• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
  We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
  arXiv  Detail & Related papers  (2024-06-25T04:27:53Z)
• Is This a Bad Table? A Closer Look at the Evaluation of Table Generation from Text [21.699434525769586]
  Existing measures for table quality evaluation fail to capture the overall semantics of the tables. We propose TabEval, a novel table evaluation strategy that captures table semantics. To validate our approach, we curate a dataset comprising of text descriptions for 1,250 diverse Wikipedia tables.
  arXiv  Detail & Related papers  (2024-06-21T02:18:03Z)
• Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis [11.746575887340798]
  Large Language Models (LLMs) have been shown to tackle table (structured) comprehension tasks without specific training.<n>We probe how in-context learning (ICL), model scale, instruction tuning, and domain bias affect Tabular QA (TQA)<n>Through an in-depth attention analysis, we reveal a strong correlation between perturbation-induced shifts in attention dispersion and performance drops.
  arXiv  Detail & Related papers  (2024-06-18T15:41:15Z)
• Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction [36.915250638481986]
  We introduce LiveSum, a new benchmark dataset for generating summary tables of competitions based on real-time commentary texts.<n>We evaluate the performances of state-of-the-art Large Language Models on this task in both fine-tuning and zero-shot settings.<n>We additionally propose a novel pipeline called $T3$(Text-Tuple-Table) to improve their performances.
  arXiv  Detail & Related papers  (2024-04-22T14:31:28Z)
• TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios [51.66718740300016]
  TableLLM is a robust large language model (LLM) with 8 billion parameters.<n>TableLLM is purpose-built for proficiently handling data manipulation tasks.<n>We have released the model checkpoint, source code, benchmarks, and a web application for user interaction.
  arXiv  Detail & Related papers  (2024-03-28T11:21:12Z)
• QTSumm: Query-Focused Summarization over Tabular Data [58.62152746690958]
  People primarily consult tables to conduct data analysis or answer specific questions. We define a new query-focused table summarization task, where text generation models have to perform human-like reasoning. We introduce a new benchmark named QTSumm for this task, which contains 7,111 human-annotated query-summary pairs over 2,934 tables.
  arXiv  Detail & Related papers  (2023-05-23T17:43:51Z)
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing [117.98107557103877]
  We present GraPPa, an effective pre-training approach for table semantic parsing. We construct synthetic question-pairs over high-free tables via a synchronous context-free grammar. To maintain the model's ability to represent real-world data, we also include masked language modeling.
  arXiv  Detail & Related papers  (2020-09-29T08:17:58Z)
• ToTTo: A Controlled Table-To-Text Generation Dataset [61.83159452483026]
  ToTTo is an open-domain English table-to-text dataset with over 120,000 training examples. We introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. While usually fluent, existing methods often hallucinate phrases that are not supported by the table.
  arXiv  Detail & Related papers  (2020-04-29T17:53:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.