Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models

• URL: http://arxiv.org/abs/2607.26000v1
• Date: Tue, 28 Jul 2026 17:16:01 GMT
• Title: Empirical Evaluation of Out-Of-Distribution Performance of Tabular Foundation Models
• Abstract summary: We present an empirical evaluation of Out-Of-Distribution (OOD) performance of nine Tabular Foundation Models (TFMs)<n>Our results show that all evaluated TFMs degrade systematically under distribution shift regardless of pre-training strategy.
• Score: 1.9917667380910824
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Tabular Foundation Models (TFMs) have emerged as novel approaches for tabular predictive tasks, demonstrating competitive predictive performance to ensemble tree-based models. Most TFMs are trained and evaluated on independent and identically distributed data, but this assumption changes in real-world scenarios due to distribution shifts, which compromise the robustness of models. Limited research has been conducted of TFMs under distribution shifts. We present an empirical evaluation of Out-Of-Distribution (OOD) performance of nine TFMs, spanning diverse pre-training strategies and architectures: TabPFNv2, TabPFNv2.5, TabPFNv2.6, TabPFNv3, TabICL, TabICLv2, Mitra, LimiX and TabFM. Three real-world datasets from the TableShift study were considered (HELOC, Voting, Childhood Lead), covering label, socioeconomic, and geographic shift types. Our results show that all evaluated TFMs degrade systematically under distribution shift regardless of pre-training strategy, with shift gaps ranging from 0.003 to 0.060 depending on shift type. The relationship between in-distribution and OOD predictive performance documented for classical tabular models extends into TFMs. We also identified a scalability gap, as high-performing models demand significant memory and computational resources beyond what standard deployment infrastructure can support. This study extends existing benchmarks for OOD in tabular data, providing evidence to support their adoption in high-stakes domains characterized by structural distribution shifts.

Related papers

• Causal Pre-training Under the Fairness Lens: An Empirical Study of TabPFN [3.059960033014892]
  We evaluate the Tabular Prior-data Fitted Network (TabPFN) and its fine-tuned variants.<n>Our results reveal that while TabPFN achieves stronger predictive accuracy compared to baselines, improvements in fairness are moderate and inconsistent.<n>These findings suggest that the causal pre-training in TabPFN is helpful but insufficient for algorithmic fairness.
  arXiv  Detail & Related papers  (2026-01-25T17:17:12Z)
• Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models [85.64873567417396]
  We introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world data.<n>Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks.
  arXiv  Detail & Related papers  (2025-10-24T07:15:06Z)
• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• Towards Fair In-Context Learning with Tabular Foundation Models [6.4989916051093815]
  We present the first investigation of fairness in Transformer-based in-context learning (ICL)<n>We evaluate three recently proposed foundation models -- TabPFNv2, TabICL, and TabDPT -- on benchmark datasets.<n>Our experiments show that the uncertainty-based strategy consistently improves group fairness metrics with minimal impact on predictive accuracy.
  arXiv  Detail & Related papers  (2025-05-14T15:53:14Z)
• TabFSBench: Tabular Benchmark for Feature Shifts in Open Environments [45.368146581808276]
  Tabular data is widely utilized in various machine learning tasks.<n>Previous research has primarily concentrated on mitigating distribution shifts, whereas feature shifts, have garnered limited attention.<n>This paper conducts the first comprehensive study on feature shifts in tabular data and introduces the first tabular feature-shift benchmark (TabFSBench)
  arXiv  Detail & Related papers  (2025-01-31T07:40:34Z)
• Fully Test-time Adaptation for Tabular Data [48.67303250592189]
  We propose the Fully Test-time Adaptation for Tabular data, which enables FTTA methods to robustly optimize the label distribution of predictions.<n>We conduct comprehensive experiments on six benchmark datasets, which are evaluated using three metrics.
  arXiv  Detail & Related papers  (2024-12-14T15:49:53Z)
• Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data [39.40116554523575]
  We present Drift-Resilient TabPFN, a fresh approach based on In-Context Learning with a Prior-Data Fitted Network. It learns to approximate Bayesian inference on synthetic datasets drawn from a prior. It improves accuracy from 0.688 to 0.744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration.
  arXiv  Detail & Related papers  (2024-11-15T23:49:23Z)
• An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation [91.62129090006745]
  This paper studies the distribution shift problem from the perspective of pre-training and data augmentation. We provide the first comprehensive empirical study focusing on pre-training and data augmentation.
  arXiv  Detail & Related papers  (2022-05-25T13:04:53Z)
• Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization [89.73665256847858]
  We show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet. We also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS.
  arXiv  Detail & Related papers  (2021-07-09T19:48:23Z)
• BREEDS: Benchmarks for Subpopulation Shift [98.90314444545204]
  We develop a methodology for assessing the robustness of models to subpopulation shift. We leverage the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity.
  arXiv  Detail & Related papers  (2020-08-11T17:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.