Aligning benchmark datasets for table structure recognition

• URL: http://arxiv.org/abs/2303.00716v2
• Date: Tue, 23 May 2023 18:57:24 GMT
• Title: Aligning benchmark datasets for table structure recognition
• Abstract summary: Benchmark datasets for table structure recognition (TSR) must be carefully processed to ensure they are consistently. In this work, we show that aligning these benchmarks$x$removing errors and inconsistency between them$x$improves performance significantly. Overall we believe our work has significant implications for benchmark processing performance and potentially other tasks as well.
• Score: 10.323063834827416
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Benchmark datasets for table structure recognition (TSR) must be carefully processed to ensure they are annotated consistently. However, even if a dataset's annotations are self-consistent, there may be significant inconsistency across datasets, which can harm the performance of models trained and evaluated on them. In this work, we show that aligning these benchmarks$\unicode{x2014}$removing both errors and inconsistency between them$\unicode{x2014}$improves model performance significantly. We demonstrate this through a data-centric approach where we adopt one model architecture, the Table Transformer (TATR), that we hold fixed throughout. Baseline exact match accuracy for TATR evaluated on the ICDAR-2013 benchmark is 65% when trained on PubTables-1M, 42% when trained on FinTabNet, and 69% combined. After reducing annotation mistakes and inter-dataset inconsistency, performance of TATR evaluated on ICDAR-2013 increases substantially to 75% when trained on PubTables-1M, 65% when trained on FinTabNet, and 81% combined. We show through ablations over the modification steps that canonicalization of the table annotations has a significantly positive effect on performance, while other choices balance necessary trade-offs that arise when deciding a benchmark dataset's final composition. Overall we believe our work has significant implications for benchmark design for TSR and potentially other tasks as well. Dataset processing and training code will be released at https://github.com/microsoft/table-transformer.

Related papers

• Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths [23.84195042261017]
  We argue that benchmark evaluation should also consider the data-centric peak performance frontier.<n>Applying this framework to the TabArena benchmark, we find that common aggregation metrics are highly correlated.
  arXiv  Detail & Related papers  (2026-08-19T13:51:05Z)
• From Diagnosis to Correction: Benchmarking and Improving Real-World Table Parsing [50.70862449978062]
  Recent documents achieve table TEDS scores above 93 on OmniDocBench v1.6, yet community feedback and our audit reveal persistent failures on complex real-world tables.<n>To quantify this gap, we introduce TableParseMap, a diagnostic benchmark of 916 real-world tables organized into five challenging scenarios and nine failure types.
  arXiv  Detail & Related papers  (2026-08-10T16:59:30Z)
• Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models [7.124188498356204]
  Tabular foundation models are pre-trained on one of three classes of corpus: curated datasets drawn from benchmark repositories, tables harvested at scale from the web, or synthetic tables sampled from a parametric generative prior.<n>In this work we take three canonical, archetypal datasets used to train tabular foundation models.<n>We characterise each corpus using aggregate features over whole tables, columns and correlations, and compare them using discriminator AUCs and k-NN coverage metrics.<n>We find that the TabICL synthetic prior occupies a narrow region of the space of real tables, that this mismatch cannot be closed by optimising prior
  arXiv  Detail & Related papers  (2026-05-07T14:29:31Z)
• TDATR: Improving End-to-End Table Recognition via Table Detail-Aware Learning and Cell-Level Visual Alignment [70.83664203825235]
  TDATR (Table Detail-Aware Table Recognition) improves end-to-end TR through table detail-aware learning and cell-level visual alignment.<n>We achieve state-of-the-art or highly competitive performance on seven benchmarks without dataset-specific fine-tuning.
  arXiv  Detail & Related papers  (2026-03-24T05:45:02Z)
• DataProphet: Demystifying Supervision Data Generalization in Multimodal LLMs [49.877224470539126]
  In this work, we take a first step toward answering a practical question: can we estimate the influence of a training dataset on a target benchmark before any training is performed?<n>Our results show that intuitive task similarity is an unreliable predictor of transferability, and that generalization depends more on the specific dataset than on its broad task category.<n>Motivated by this finding, we propose DATAPROPHET, a training-free metric that combines multimodal perplexity, similarity, and data diversity.
  arXiv  Detail & Related papers  (2026-03-20T06:42:26Z)
• TabSieve: Explicit In-Table Evidence Selection for Tabular Prediction [14.270578219134997]
  We propose TabSieve, a select-then-predict framework that makes evidence usage explicit and auditable.<n>Given a table and a query row, TabSieve first selects a small set of informative rows as evidence and then predicts the missing target conditioned on the selected evidence.<n>Experiments on a held-out benchmark of 75 classification and 52 regression tables show that TabSieve consistently improves performance across shot budgets.
  arXiv  Detail & Related papers  (2026-02-12T08:28:58Z)
• JT-DA: Enhancing Data Analysis with Tool-Integrated Table Reasoning Large Language Models [58.408398005993455]
  JT-DA-8B is a specialized large language model designed for complex table reasoning tasks across diverse real-world scenarios.<n>We construct a comprehensive and diverse training corpus with 34 well-defined table reasoning tasks, by aggregating 29 public table QA datasets and 3 million tables.<n> Experimental results show that JT-DA-8B achieves strong performance in various table reasoning tasks.
  arXiv  Detail & Related papers  (2025-12-07T14:29:23Z)
• SAVeD: Semantic Aware Version Discovery [6.220725124423482]
  SAVeD is a contrastive learning-based framework for identifying versions of structured datasets.<n>Our experiments span five canonical datasets from the Semantic Versioning in Databases Benchmark.
  arXiv  Detail & Related papers  (2025-11-21T15:11:15Z)
• Advancing Retrieval-Augmented Generation for Structured Enterprise and Internal Data [0.0]
  Large Language Models (LLMs) have strong generative capabilities.<n>They are limited by static pretraining, short context windows, and challenges in processing heterogeneous data formats.<n>Conventional Retrieval-Augmented Generation (RAG) frameworks address some of these gaps but often struggle with structured and semi-structured data.<n>This work proposes an advanced RAG framework that combines hybrid retrieval strategies using dense embeddings (all-mpnet-base-v2) and BM25, enhanced by metadata-aware filtering with SpaCy NER and cross-encoder reranking.
  arXiv  Detail & Related papers  (2025-07-16T17:13:06Z)
• The Mighty ToRR: A Benchmark for Table Reasoning and Robustness [45.420943398134845]
  ToRR is a benchmark for Table Reasoning and Robustness, measuring model performance and robustness on table-related tasks. We present a leaderboard as well as comprehensive analyses of the results of leading models over ToRR.
  arXiv  Detail & Related papers  (2025-02-26T18:56:38Z)
• Towards Better Understanding Table Instruction Tuning: Decoupling the Effects from Data versus Models [62.47618742274461]
  We fine-tune base models from the Mistral, OLMo, and Phi families on existing public training datasets. Our replication achieves performance on par with or surpassing existing table LLMs. We decouple the contributions of training data and the base model, providing insight into their individual impacts.
  arXiv  Detail & Related papers  (2025-01-24T18:50:26Z)
• KET-QA: A Dataset for Knowledge Enhanced Table Question Answering [63.56707527868466]
  We propose to use a knowledge base (KB) as the external knowledge source for TableQA. Every question requires the integration of information from both the table and the sub-graph to be answered. We design a retriever-reasoner structured pipeline model to extract pertinent information from the vast knowledge sub-graph.
  arXiv  Detail & Related papers  (2024-05-13T18:26:32Z)
• 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on Relational DBs [67.47600679176963]
  RDBs store vast amounts of rich, informative data spread across interconnected tables. The progress of predictive machine learning models falls behind advances in other domains such as computer vision or natural language processing. We explore a class of baseline models predicated on converting multi-table datasets into graphs. We assemble a diverse collection of large-scale RDB datasets and (ii) coincident predictive tasks.
  arXiv  Detail & Related papers  (2024-04-28T15:04:54Z)
• Breaking Free Transformer Models: Task-specific Context Attribution Promises Improved Generalizability Without Fine-tuning Pre-trained LLMs [1.5138606851862884]
  We present a framework that allows for maintaining generalizability and enhances the performance on the downstream task. We show that a linear transformation of the text representation from any transformer model using the task-specific concept operator results in a projection onto the latent concept space. Experimental results on three datasets, namely HateXplain, IMDB reviews, and Social Media Attributions, illustrate that the proposed model attains superior accuracy and generalizability.
  arXiv  Detail & Related papers  (2024-01-30T00:23:29Z)
• SMaRt: Improving GANs with Score Matching Regularity [94.81046452865583]
  Generative adversarial networks (GANs) usually struggle in learning from highly diverse data, whose underlying manifold is complex. We show that score matching serves as a promising solution to this issue thanks to its capability of persistently pushing the generated data points towards the real data manifold. We propose to improve the optimization of GANs with score matching regularity (SMaRt)
  arXiv  Detail & Related papers  (2023-11-30T03:05:14Z)
• Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule [55.08778142798106]
  We propose auxiliary tasks that exploit the alignment between the original and corrected sentences. We formulate each task as a sequence-to-sequence problem and perform multi-task training. We find that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance.
  arXiv  Detail & Related papers  (2023-11-20T14:50:12Z)
• Benchmarking a Benchmark: How Reliable is MS-COCO? [0.0]
  Sama-COCO, a re-annotation of MS-COCO, is used to discover potential biases by leveraging a shape analysis pipeline. A model is trained and evaluated on both datasets to examine the impact of different annotation conditions.
  arXiv  Detail & Related papers  (2023-11-05T16:55:40Z)
• Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation [67.30538142519067]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM) A deep neural network is then trained to associate these meta-representations with dataset-specific classification confidences. Experiments validate that TabPTM achieves promising performance in new datasets, even under few-shot scenarios.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• Finetuning BERT on Partially Annotated NER Corpora [0.0]
  This work presents an approach to finetuning BERT on partially labelled datasets using self-supervision and label preprocessing. Our approach outperforms the previous LSTM-based label preprocessing baseline, significantly improving the performance on poorly labelled datasets.
  arXiv  Detail & Related papers  (2022-11-25T19:54:30Z)
• The Document Vectors Using Cosine Similarity Revisited [0.48733623015338234]
  The current state-of-the-art test accuracy (97.42%) on the IMDB movie reviews dataset was reported by citetthongtan-phienthrakul 2019-sentiment. We show that the previously reported test accuracy of 97.42% is invalid and should be corrected to 93.68%.
  arXiv  Detail & Related papers  (2022-05-26T13:36:54Z)
• Understanding tables with intermediate pre-training [11.96734018295146]
  We adapt TAPAS, a table-based BERT model, to recognize entailment. We evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency.
  arXiv  Detail & Related papers  (2020-10-01T17:43:27Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.