Is This a Bad Table? A Closer Look at the Evaluation of Table Generation from Text

• URL: http://arxiv.org/abs/2406.14829v3
• Date: Sat, 23 Nov 2024 19:08:45 GMT
• Title: Is This a Bad Table? A Closer Look at the Evaluation of Table Generation from Text
• Authors: Pritika Ramu, Aparna Garimella, Sambaran Bandyopadhyay,
• Abstract summary: Existing measures for table quality evaluation fail to capture the overall semantics of the tables. We propose TabEval, a novel table evaluation strategy that captures table semantics. To validate our approach, we curate a dataset comprising of text descriptions for 1,250 diverse Wikipedia tables.
• Score: 21.699434525769586
• License:
• Abstract: Understanding whether a generated table is of good quality is important to be able to use it in creating or editing documents using automatic methods. In this work, we underline that existing measures for table quality evaluation fail to capture the overall semantics of the tables, and sometimes unfairly penalize good tables and reward bad ones. We propose TabEval, a novel table evaluation strategy that captures table semantics by first breaking down a table into a list of natural language atomic statements and then compares them with ground truth statements using entailment-based measures. To validate our approach, we curate a dataset comprising of text descriptions for 1,250 diverse Wikipedia tables, covering a range of topics and structures, in contrast to the limited scope of existing datasets. We compare TabEval with existing metrics using unsupervised and supervised text-to-table generation methods, demonstrating its stronger correlation with human judgments of table quality across four datasets.

Related papers

• ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models [58.34560740973768]
  We introduce a framework that leverages language models (LMs) to generate literature review tables. A new dataset of 2,228 literature review tables extracted from ArXiv papers synthesize a total of 7,542 research papers. We evaluate LMs' abilities to reconstruct reference tables, finding this task benefits from additional context.
  arXiv  Detail & Related papers  (2024-10-25T18:31:50Z)
• UniTabNet: Bridging Vision and Language Models for Enhanced Table Structure Recognition [55.153629718464565]
  We introduce UniTabNet, a novel framework for table structure parsing based on the image-to-text model. UniTabNet employs a divide-and-conquer'' strategy, utilizing an image-to-text model to decouple table cells and integrating both physical and logical decoders to reconstruct the complete table structure.
  arXiv  Detail & Related papers  (2024-09-20T01:26:32Z)
• TDeLTA: A Light-weight and Robust Table Detection Method based on Learning Text Arrangement [34.73880086005418]
  We propose a novel, light-weighted and robust Table Detection method based on Learning Text Arrangement, namely TDeLTA. To locate the tables precisely, we design a text-classification task, classifying the text blocks into 4 categories according to their semantic roles in the tables. Compared to several state-of-the-art methods, TDeLTA achieves competitive results with only 3.1M model parameters on the large-scale public datasets.
  arXiv  Detail & Related papers  (2023-12-18T09:18:43Z)
• Towards Table-to-Text Generation with Pretrained Language Model: A Table Structure Understanding and Text Deliberating Approach [60.03002572791552]
  We propose a table structure understanding and text deliberating approach, namely TASD. Specifically, we devise a three-layered multi-head attention network to realize the table-structure-aware text generation model. Our approach can generate faithful and fluent descriptive texts for different types of tables.
  arXiv  Detail & Related papers  (2023-01-05T14:03:26Z)
• TableFormer: Robust Transformer Modeling for Table-Text Encoding [18.00127368618485]
  Existing models for table understanding require linearization of the table structure, where row or column order is encoded as an unwanted bias. In this work, we propose a robust and structurally aware table-text encoding architecture TableFormer.
  arXiv  Detail & Related papers  (2022-03-01T07:23:06Z)
• Learning Better Representation for Tables by Self-Supervised Tasks [23.69766883380125]
  We propose two self-supervised tasks, Number Ordering and Significance Ordering, to help to learn better table representation. We test our methods on the widely used dataset ROTOWIRE which consists of NBA game statistic and related news.
  arXiv  Detail & Related papers  (2020-10-15T09:03:38Z)
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing [117.98107557103877]
  We present GraPPa, an effective pre-training approach for table semantic parsing. We construct synthetic question-pairs over high-free tables via a synchronous context-free grammar. To maintain the model's ability to represent real-world data, we also include masked language modeling.
  arXiv  Detail & Related papers  (2020-09-29T08:17:58Z)
• Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints [63.84063384518667]
  We propose a novel Transformer-based generation framework to achieve the goal. Core techniques in our method to enforce faithfulness include a new table-text optimal-transport matching loss. To evaluate faithfulness, we propose a new automatic metric specialized to the table-to-text generation problem.
  arXiv  Detail & Related papers  (2020-05-03T02:54:26Z)
• ToTTo: A Controlled Table-To-Text Generation Dataset [61.83159452483026]
  ToTTo is an open-domain English table-to-text dataset with over 120,000 training examples. We introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. While usually fluent, existing methods often hallucinate phrases that are not supported by the table.
  arXiv  Detail & Related papers  (2020-04-29T17:53:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.