Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression

• URL: http://arxiv.org/abs/2603.26611v1
• Date: Fri, 27 Mar 2026 17:07:21 GMT
• Title: Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression
• Abstract summary: Conditional density estimation is essential in settings with heteroscedasticity, multimodality, or asymmetric uncertainty.<n>Tablet foundation models, such as TabPFN and TabICL, naturally produce predictive distributions, but their effectiveness as general-purpose CDE methods has not been systematically evaluated.<n>We benchmark three model variants against a diverse set of parametric, tree-based, and neural CDE baselines on 39 real-world datasets.
• Score: 2.4816602844974835
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conditional density estimation (CDE) - recovering the full conditional distribution of a response given tabular covariates - is essential in settings with heteroscedasticity, multimodality, or asymmetric uncertainty. Recent tabular foundation models, such as TabPFN and TabICL, naturally produce predictive distributions, but their effectiveness as general-purpose CDE methods has not been systematically evaluated, unlike their performance for point prediction, which is well studied. We benchmark three tabular foundation model variants against a diverse set of parametric, tree-based, and neural CDE baselines on 39 real-world datasets, across training sizes from 50 to 20,000, using six metrics covering density accuracy, calibration, and computation time. Across all sample sizes, foundation models achieve the best CDE loss, log-likelihood, and CRPS on the large majority of datasets tested. Calibration is competitive at small sample sizes but, for some metrics and datasets, lags behind task-specific neural baselines at larger sample sizes, suggesting that post-hoc recalibration may be a valuable complement. In a photometric redshift case study using SDSS DR18, TabPFN exposed to 50,000 training galaxies outperforms all baselines trained on the full 500,000-galaxy dataset. Taken together, these results establish tabular foundation models as strong off-the-shelf conditional density estimators.

Related papers

• Balanced Adaptive Prototype Selection for Scalable TabPFN Inference on Large-Scale Tabular Data [1.9768907237963547]
  This paper introduces Balanced Adaptive Prototype Selection (BAPS), a framework for constructing compact, information-preserving contexts for scalable TabPFN inference.<n> Experiments on the million-row HIGGS and SUSY datasets show that 512 prototypes retain strong predictive performance and reliable calibration.
  arXiv  Detail & Related papers  (2026-08-13T09:12:02Z)
• Sequential Structure in Intraday Futures Data: LSTM vs Gradient Boosting on MNQ [0.0]
  We test whether five-minute OHLCV bar sequences contain exploitable sequential predictive structure at the scale of a single instrument dataset.<n>Using 944 trading days from 2021-2025, four model configurations are evaluated under strict expanding-window walk-forward validation.<n>No configuration produces statistically significant out-of-sample accuracy above the 51.8% base rate.
  arXiv  Detail & Related papers  (2026-05-18T01:03:28Z)
• Scalable Learning of Multivariate Distributions via Coresets [48.02318786960811]
  We develop a novel coreset construction for multivariate conditional transformation models (MCTMs) to enhance their scalability and training efficiency.<n>Our approach yields substantial data reduction via importance sampling.<n>Compared to conventional full-parametric models, where coresets have been incorporated before, our semi-parametric approach exhibits enhanced adaptability.
  arXiv  Detail & Related papers  (2026-03-20T09:28:43Z)
• TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis [22.883505574924303]
  We propose TabularMath, a diagnostic benchmark of 114 deterministic problems (233,472 rows) generated from verified programs based on 8K and AIME.<n>On standard regression metrics, TabPFN v2.5 performs remarkably well, achieving R2=0.998 in-distribution and maintaining positive R2 even under distribution shift.<n>When we measure rounded consistency (exact integer accuracy), a different picture emerges: TabPFN v2.5 drops below 10% on out-of-distribution data, while ICL maintains around 40%.
  arXiv  Detail & Related papers  (2026-01-25T23:44:03Z)
• A recursive Bayesian neural network for constitutive modeling of sands under monotonic and cyclic loading [0.0]
  In engineering, models are central to capturing soil behavior across diverse drainage conditions, stress paths,and loading histories.<n>This study introduces a recursive Bayesian neural network (rBNN) framework that unifies temporal sequence learning with generalized inference.<n>The framework is validated against four datasets spanning both simulated and experimental triaxial tests.
  arXiv  Detail & Related papers  (2025-01-17T10:15:03Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• Extrapolation to complete basis-set limit in density-functional theory by quantile random-forest models [0.41104099603771493]
  We find a machine-learning model that extrapolates finite basis-size calculations to the CBS limit. A quantile-random-forest model is used to estimate the total-energy correction with respect to a fully converged calculation. Our approach also provides prediction intervals, which quantify the uncertainty of the models' predictions.
  arXiv  Detail & Related papers  (2023-03-26T15:43:37Z)
• Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning [78.30395044401321]
  We develop a novel model-based approach to reinforcement learning (MBRL) It relaxes the assumptions on the target transition model to belong to a generic family of mixture models. It can achieve up-to 50 percent reduction in wall clock time in some continuous control environments.
  arXiv  Detail & Related papers  (2022-06-02T17:27:49Z)
• Bayesian Low-Rank Interpolative Decomposition for Complex Datasets [4.913248451323163]
  We introduce a probabilistic model for learning interpolative decomposition (ID), which is commonly used for feature selection, low-rank approximation, and identifying hidden patterns in data. We evaluate the model on a variety of real-world datasets including CCLE EC50, CCLE IC50, CTRP EC50,and MovieLens 100K datasets with different sizes, and dimensions.
  arXiv  Detail & Related papers  (2022-05-30T03:06:48Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)
• Improving Maximum Likelihood Training for Text Generation with Density Ratio Estimation [51.091890311312085]
  We propose a new training scheme for auto-regressive sequence generative models, which is effective and stable when operating at large sample space encountered in text generation. Our method stably outperforms Maximum Likelihood Estimation and other state-of-the-art sequence generative models in terms of both quality and diversity.
  arXiv  Detail & Related papers  (2020-07-12T15:31:24Z)
• TraDE: Transformers for Density Estimation [101.20137732920718]
  TraDE is a self-attention-based architecture for auto-regressive density estimation. We present a suite of tasks such as regression using generated samples, out-of-distribution detection, and robustness to noise in the training data.
  arXiv  Detail & Related papers  (2020-04-06T07:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.