Causal Pre-training Under the Fairness Lens: An Empirical Study of TabPFN

• URL: http://arxiv.org/abs/2601.17912v2
• Date: Tue, 27 Jan 2026 11:11:17 GMT
• Title: Causal Pre-training Under the Fairness Lens: An Empirical Study of TabPFN
• Authors: Qinyi Liu, Mohammad Khalil, Naman Goel,
• Abstract summary: We evaluate the Tabular Prior-data Fitted Network (TabPFN) and its fine-tuned variants.<n>Our results reveal that while TabPFN achieves stronger predictive accuracy compared to baselines, improvements in fairness are moderate and inconsistent.<n>These findings suggest that the causal pre-training in TabPFN is helpful but insufficient for algorithmic fairness.
• Score: 3.059960033014892
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Foundation models for tabular data, such as the Tabular Prior-data Fitted Network (TabPFN), are pre-trained on a massive number of synthetic datasets generated by structural causal models (SCM). They leverage in-context learning to offer high predictive accuracy in real-world tasks. However, the fairness properties of these foundational models, which incorporate ideas from causal reasoning during pre-training, remain underexplored. In this work, we conduct a comprehensive empirical evaluation of TabPFN and its fine-tuned variants, assessing predictive performance, fairness, and robustness across varying dataset sizes and distributional shifts. Our results reveal that while TabPFN achieves stronger predictive accuracy compared to baselines and exhibits robustness to spurious correlations, improvements in fairness are moderate and inconsistent, particularly under missing-not-at-random (MNAR) covariate shifts. These findings suggest that the causal pre-training in TabPFN is helpful but insufficient for algorithmic fairness, highlighting implications for deploying TabPFN (and similar) models in practice and the need for further fairness interventions.

Related papers

• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• Tabular foundation model for GEOAI benchmark problems BM/AirportSoilProperties/2/2025 [2.07098502859192]
  This paper presents a novel application of the Tabular Prior-Data Fitted Network (TabPFN) to site characterization problems defined in the GEOAI benchmark BM/AirportSoilProperties/2/2025.<n>We apply TabPFN in a zero-training, few-shot, in-spatial learning setting and provide it with additional context from the big indirect database (BID)<n>The study demonstrates that TabPFN, as a general-purpose foundation model, achieved superior accuracy and well-calibrated predictive distributions.
  arXiv  Detail & Related papers  (2025-09-03T10:21:18Z)
• Multiply Robust Conformal Risk Control with Coarsened Data [0.0]
  Conformal Prediction (CP) has recently received a tremendous amount of interest.<n>In this paper, we consider the general problem of obtaining distribution-free valid prediction regions for an outcome given coarsened data.<n>Our principled use of semiparametric theory has the key advantage of facilitating flexible machine learning methods.
  arXiv  Detail & Related papers  (2025-08-21T12:14:44Z)
• Towards Fair In-Context Learning with Tabular Foundation Models [6.4989916051093815]
  We present the first investigation of fairness in Transformer-based in-context learning (ICL)<n>We evaluate three recently proposed foundation models -- TabPFNv2, TabICL, and TabDPT -- on benchmark datasets.<n>Our experiments show that the uncertainty-based strategy consistently improves group fairness metrics with minimal impact on predictive accuracy.
  arXiv  Detail & Related papers  (2025-05-14T15:53:14Z)
• Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models [14.125143586947177]
  We show how TabPFN can be used as pre-trained autoregressive conditional density estimators for SBI.<n>NPE-PFN eliminates the need for inference network selection, training, and hyper parameter tuning.<n>It exhibits superior robustness to model misspecification and can be scaled to simulation budgets that exceed the context size limit of TabPFN.
  arXiv  Detail & Related papers  (2025-04-24T15:29:39Z)
• EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Networks [55.214444066134114]
  We design a fully target-equivariant architecture-ensuring permutation invariance via equivariant encoders, decoders, and a bi-attention mechanism.<n> Empirical evaluation on standard classification benchmarks shows that, on datasets with more classes than those seen during pre-training, our model matches or surpasses existing methods while incurring lower computational overhead.
  arXiv  Detail & Related papers  (2025-02-10T17:11:20Z)
• A recursive Bayesian neural network for constitutive modeling of sands under monotonic and cyclic loading [0.0]
  In engineering, models are central to capturing soil behavior across diverse drainage conditions, stress paths,and loading histories.<n>This study introduces a recursive Bayesian neural network (rBNN) framework that unifies temporal sequence learning with generalized inference.<n>The framework is validated against four datasets spanning both simulated and experimental triaxial tests.
  arXiv  Detail & Related papers  (2025-01-17T10:15:03Z)
• Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Data [39.40116554523575]
  We present Drift-Resilient TabPFN, a fresh approach based on In-Context Learning with a Prior-Data Fitted Network. It learns to approximate Bayesian inference on synthetic datasets drawn from a prior. It improves accuracy from 0.688 to 0.744 and ROC AUC from 0.786 to 0.832 while maintaining stronger calibration.
  arXiv  Detail & Related papers  (2024-11-15T23:49:23Z)
• Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs [10.494477811252034]
  Fine-tuning multiplicity can arise in Tabular LLMs on classification tasks.<n>Our work formalizes this unique challenge of fine-tuning multiplicity in Tabular LLMs.<n>We propose a novel measure to quantify consistency of individual predictions without expensive model retraining.
  arXiv  Detail & Related papers  (2024-07-04T22:22:09Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• When Rigidity Hurts: Soft Consistency Regularization for Probabilistic Hierarchical Time Series Forecasting [69.30930115236228]
  Probabilistic hierarchical time-series forecasting is an important variant of time-series forecasting. Most methods focus on point predictions and do not provide well-calibrated probabilistic forecasts distributions. We propose PROFHiT, a fully probabilistic hierarchical forecasting model that jointly models forecast distribution of entire hierarchy.
  arXiv  Detail & Related papers  (2023-10-17T20:30:16Z)
• Causality-oriented robustness: exploiting general noise interventions [4.64479351797195]
  In this paper, we focus on causality-oriented robustness and propose Distributional Robustness via Invariant Gradients (DRIG)<n>DRIG exploits general noise interventions in training data for robust predictions against unseen interventions.<n>We show that our framework includes anchor regression as a special case, and that it yields prediction models that protect against more diverse perturbations.
  arXiv  Detail & Related papers  (2023-07-18T16:22:50Z)
• Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach [72.19525160912943]
  We first theoretically demonstrate the inherent connection between distribution shift, data perturbation, and model weight perturbation. We then analyze the sufficient conditions to guarantee fairness for the target dataset. Motivated by these sufficient conditions, we propose robust fairness regularization (RFR)
  arXiv  Detail & Related papers  (2023-03-06T17:19:23Z)
• Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions [59.284907093349425]
  Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. We provide a language for describing how training data influences predictions, through a causal framework. Our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone.
  arXiv  Detail & Related papers  (2022-07-28T17:36:24Z)
• When Rigidity Hurts: Soft Consistency Regularization for Probabilistic Hierarchical Time Series Forecasting [69.30930115236228]
  Probabilistic hierarchical time-series forecasting is an important variant of time-series forecasting. Most methods focus on point predictions and do not provide well-calibrated probabilistic forecasts distributions. We propose PROFHiT, a fully probabilistic hierarchical forecasting model that jointly models forecast distribution of entire hierarchy.
  arXiv  Detail & Related papers  (2022-06-16T06:13:53Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Elastic weight consolidation for better bias inoculation [24.12790037712358]
  Elastic weight consolidation (EWC) allows fine-tuning of models to mitigate biases. EWC dominates standard fine-tuning, yielding models with lower levels of forgetting on the original (biased) dataset.
  arXiv  Detail & Related papers  (2020-04-29T17:45:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.