Understanding the Surprising Generalization Properties of Tabular Foundation Models

• URL: http://arxiv.org/abs/2608.17957v1
• Date: Tue, 18 Aug 2026 16:11:08 GMT
• Title: Understanding the Surprising Generalization Properties of Tabular Foundation Models
• Abstract summary: We show that surprisingly strong transfer can emerge from self-grained pre-training on just a single real table.<n>We show that the same task-centric perspective can help corpus design at scale.
• Score: 39.6543292456356
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.

Related papers

• Beyond IID: How General Are Tabular Foundation Models, Really? [76.03669797389378]
  Research communities are increasingly evaluating foundation models on diverse datasets and tasks.<n>Standard benchmarks are mostly defined for tasks where tabular foundation models already excel.<n>We introduce BeyondArena, the first unified holistic benchmark for tabular data.
  arXiv  Detail & Related papers  (2026-06-29T14:55:52Z)
• MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image [52.47434184153733]
  We introduce MulTaBench, a benchmark of 40 datasets split equally between image-tabular and text-tabular tasks.<n>We show that the gains from target-aware representation tuning generalize across both text and image modalities.
  arXiv  Detail & Related papers  (2026-05-11T14:12:05Z)
• Generalization Can Emerge in Tabular Foundation Models From a Single Table [38.07740881271672]
  We show that simple self-supervised pre-training on just a emphsingle real table can produce surprisingly strong transfer across heterogeneous benchmarks.<n>We then connect this to the pre-training procedure shared by most TFMs and show that the number and quality of emphtasks one can construct from a dataset is key to downstream performance.
  arXiv  Detail & Related papers  (2025-11-12T19:12:40Z)
• Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models [85.64873567417396]
  We introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world data.<n>Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks.
  arXiv  Detail & Related papers  (2025-10-24T07:15:06Z)
• Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models [64.02612380298228]
  Recent studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT)<n>We propose a model in which TTT achieves a substantially smaller in-distribution test error than global training.<n>We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet.
  arXiv  Detail & Related papers  (2025-09-29T09:24:52Z)
• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• Table Foundation Models: on knowledge pre-training for tabular learning [47.485516405457595]
  TARTE is a foundation model that transforms tables to knowledge-enhanced vector representations using the string.<n>Pre-trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost.
  arXiv  Detail & Related papers  (2025-05-20T14:27:51Z)
• Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard [27.224577475861214]
  Tabular data, a fundamental data format in machine learning, is predominantly utilized in competitions and real-world applications.<n>We propose an in-context ensemble framework for tabular prediction that leverages large language models.<n>Our method constructs a context around each test instance using its nearest neighbors and the predictions from a pool of external models.
  arXiv  Detail & Related papers  (2025-05-19T17:52:58Z)
• Representation Learning for Tabular Data: A Comprehensive Survey [23.606506938919605]
  Tabular data, structured as rows and columns, is among the most prevalent data types in machine learning classification and regression applications.<n>Deep Neural Networks (DNNs) have recently demonstrated promising results through their capability of representation learning.<n>We organize existing methods into three main categories according to their generalization capabilities.
  arXiv  Detail & Related papers  (2025-04-17T17:58:23Z)
• Rethinking Table Instruction Tuning [29.139828718538418]
  We evaluate abilities in existing table LLMs and find significant declines in both out-of-domain table understanding and general capabilities.<n>We introduce TAMA, a TAble LLM instruction-tuned from LLaMA 3.1 8B Instruct, which achieves performance on par with, or surpassing GPT-3.5 and GPT-4 on table tasks.
  arXiv  Detail & Related papers  (2025-01-24T18:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.