Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models

• URL: http://arxiv.org/abs/2605.06343v1
• Date: Thu, 07 May 2026 14:29:31 GMT
• Title: Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models
• Abstract summary: Tabular foundation models are pre-trained on one of three classes of corpus: curated datasets drawn from benchmark repositories, tables harvested at scale from the web, or synthetic tables sampled from a parametric generative prior.<n>In this work we take three canonical, archetypal datasets used to train tabular foundation models.<n>We characterise each corpus using aggregate features over whole tables, columns and correlations, and compare them using discriminator AUCs and k-NN coverage metrics.<n>We find that the TabICL synthetic prior occupies a narrow region of the space of real tables, that this mismatch cannot be closed by optimising prior
• Score: 7.124188498356204
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Tabular foundation models are pre-trained on one of three classes of corpus: curated datasets drawn from benchmark repositories, tables harvested at scale from the web, or synthetic tables sampled from a parametric generative prior. Despite the centrality of pre-training data to model performance, little is known about how these corpora relate to one another in distribution, and the impact this has on downstream performance. In this work we take three canonical, archetypal datasets used to train tabular foundation models; the T4 dataset represents web-scraped corpora, the TabFM dataset curated tables from Kaggle, and the TabICL dataset as the only well-used synthetic prior with publicly available parameters. We characterise each corpus using aggregate features over whole tables, columns and correlations, and compare them using discriminator AUCs and k-NN coverage metrics. We find that the TabICL synthetic prior occupies a narrow region of the space of real tables, that this mismatch cannot be closed by optimising prior hyper-parameters across more than 86 thousand configurations, and that curated and web-scraped corpora are broadly interchangeable on a distributional level in feature space. Surprisingly, the distributional gap between synthetic pre-training data and real tables has a clearly detectable effect on performance under neither feature-based proximity measures or TabICL's own internal representations, suggesting that coverage of the real-data distribution is not the primary driver of TabICL's generalisation.

Related papers

• Revisiting Metafeatures to Explain Model Differences on Tabular Data [12.445205798094412]
  Using the TabArena benchmark results, we analyze dataset-level performance gaps and relate them to model-agnostic dataset descriptors.<n>We find that for neural network vs. tree gaps, no meta-feature survives false discovery control, and one association is robust but does not generalize when tested in leave-one-dataset-out prediction.
  arXiv  Detail & Related papers  (2026-05-27T12:50:22Z)
• Tabular Foundation Model for Generative Modelling [34.3599321018728]
  Generative modelling requires robust, holistic representation learning for a given data modality, rather than optimisation for a supervised prediction target alone.<n>Existing tabular foundation generators have not yet consistently matched strong dataset-specific generators in synthetic data quality.<n>We introduce TabFORGE, built on pretrained textbfTabular textbfFOundational textbfRepresentations for textbfGEneration.
  arXiv  Detail & Related papers  (2026-05-10T08:52:28Z)
• Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms [0.0]
  Tabular foundation models (TFMs) generalize across heterogeneous datasets through in-context learning (ICL)<n>We study TabPFN and its attention mechanisms for binary classification problems with controlled synthetic perturbations.<n>We analyze internal signals including attention concentration and attention-based feature ranking metrics.
  arXiv  Detail & Related papers  (2026-04-06T17:16:37Z)
• TableNet A Large-Scale Table Dataset with LLM-Powered Autonomous [4.207642146162365]
  Table Structure Recognition (TSR) requires the logical reasoning ability of large language models (LLMs) to handle complex table layouts.<n>We present TableNet dataset, a new table structure recognition dataset collected and generated through multiple sources.
  arXiv  Detail & Related papers  (2026-02-27T02:44:38Z)
• RDBLearn: Simple In-Context Prediction Over Relational Databases [21.996337463952255]
  We show that a simple recipe can be extended to relational prediction with a simple recipe.<n>We package this approach in textitRDBLearn, an easy-to-use toolkit with a scikit-learn-style estimator interface.<n>Across a broad collection of RelBench and 4DBInfer datasets, RDBLearn is the best-performing foundation model approach we evaluate.
  arXiv  Detail & Related papers  (2026-02-14T09:24:04Z)
• PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models [51.42043158297229]
  We introduce Pluel, a framework to synthesize multi-tabular relational databases from scratch.<n>In a step-by-step fashion, Pluel models (1) schemas with directed graphs, (2) inter-table primary-foreign key connectivity with bipartite graphs, and, (3) feature distributions in tables via conditional causal mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T21:35:18Z)
• Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models [85.64873567417396]
  We introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world data.<n>Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks.
  arXiv  Detail & Related papers  (2025-10-24T07:15:06Z)
• LLM-TabLogic: Preserving Inter-Column Logical Relationships in Synthetic Tabular Data via Prompt-Guided Latent Diffusion [49.898152180805454]
  Synthetic datasets must maintain domain-specific logical consistency.<n>Existing generative models often overlook these inter-column relationships.<n>This study presents the first method to effectively preserve inter-column relationships without requiring domain knowledge.
  arXiv  Detail & Related papers  (2025-03-04T00:47:52Z)
• TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation [91.50296404732902]
  We introduce TabDiff, a joint diffusion framework that models all mixed-type distributions of tabular data in one model.<n>Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data.<n>TabDiff achieves superior average performance over existing competitive baselines, with up to $22.5%$ improvement over the state-of-the-art model on pair-wise column correlation estimations.
  arXiv  Detail & Related papers  (2024-10-27T22:58:47Z)
• LaTable: Towards Large Tabular Models [63.995130144110156]
  Tabular generative foundation models are hard to build due to the heterogeneous feature spaces of different datasets. LaTable is a novel diffusion model that addresses these challenges and can be trained across different datasets. We find that LaTable outperforms baselines on in-distribution generation, and that finetuning LaTable can generate out-of-distribution datasets better with fewer samples.
  arXiv  Detail & Related papers  (2024-06-25T16:03:50Z)
• Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective [71.45945607871715]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM)<n>The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels.<n>Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing [117.98107557103877]
  We present GraPPa, an effective pre-training approach for table semantic parsing. We construct synthetic question-pairs over high-free tables via a synchronous context-free grammar. To maintain the model's ability to represent real-world data, we also include masked language modeling.
  arXiv  Detail & Related papers  (2020-09-29T08:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.