Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules

• URL: http://arxiv.org/abs/2603.08206v3
• Date: Mon, 16 Mar 2026 14:03:47 GMT
• Title: Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules
• Abstract summary: We show that proper scoring rules induce different model rankings and different inductive biases during training.<n>Fine-tuning realTabPFNv2.5 with scoring rules not seen during pretraining yields consistent improvements on the corresponding metrics.
• Score: 0.7009487789080344
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular foundation models such as TabPFN and TabICL already produce full predictive distributions, yet the benchmarks used to evaluate them (TabArena, TALENT, and others) still rely almost exclusively on point-estimate metrics (RMSE, $R^2$). This mismatch implicitly rewards models that elicit a good conditional mean while ignoring the quality of the predicted distribution. We make two contributions. First, we propose supplementing standard point metrics with proper scoring rules (CRPS, CRLS, and the Interval Score) and provide a head-to-head comparison of realTabPFNv2.5 and TabICLv2 with regards to some proper scoring rules across 20 OpenML regression datasets. Second, we show analytically and empirically that different proper scoring rules induce different model rankings and different inductive biases during training, even though each rule is individually minimized by the true distribution. Fine-tuning realTabPFNv2.5 with scoring rules not seen during pretraining (CRLS, $β=1.8$ energy score) yields consistent improvements on the corresponding metrics, confirming that the training loss shapes the model beyond what propriety alone guarantees. Together, these findings argue for (i) reporting distributional metrics in tabular regression benchmarks and (ii) making the training objective of foundation models adaptable (via fine-tuning or task-token conditioning) to the scoring rule relevant to the downstream decision problem.

Related papers

• Do Tabular Foundation Models Agree with Themselves? [9.642976236410833]
  We propose asking a different question: could a model's predictions result from any joint distribution?<n>The first is marginalization consistency, which demands that marginalized conditionals are equal to directly predicted marginals.<n>The second is factorization consistency, which demands that different factorization orders result in equal joint distributions.
  arXiv  Detail & Related papers  (2026-08-06T13:09:54Z)
• Beyond Accuracy: Measuring Logical Compliance of Predictive Models [2.963223599781967]
  Rule Violation Score (RVS) is a complementary evaluation metric that quantifies the extent to which a predictive model respects a given set of logical rules.<n>We evaluate RVS on three benchmarks covering knowledge graph link prediction and relational regression.<n>Our results demonstrate that two models achieving comparable predictive accuracy can exhibit substantially different levels of logical compliance.
  arXiv  Detail & Related papers  (2026-06-18T13:21:04Z)
• CRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batching [1.4211459456330362]
  Prior-fitted networks (PFNs) are a promising class of foundation models that perform in-context learning.<n>We propose CRUMB (Clustered Retrieval Using Minimised-MMD ), a three-stage inference wrapper that clusters the test queries.<n>CRUMB selects a small, distributionally matched training subset for each cluster by greedily minimising the maximum mean discrepancy.
  arXiv  Detail & Related papers  (2026-06-09T22:07:04Z)
• PriFT: Prior-Support Guided Supervised Fine-Tuning [74.65198014829393]
  Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation.<n>SFT fits fixed demonstrations token by token, including targets poorly aligned with the model's pretrained distribution.<n>We propose PriFT, which derives token weights from a frozen pretrained reference to obtain a stable reweighting signal unaffected by fine-tuning.
  arXiv  Detail & Related papers  (2026-06-08T12:14:06Z)
• ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules [0.7009487789080344]
  Tabular foundation models such as TabPFN and TabICL already produce full distributions yet prevailing regression benchmarks evaluate them almost exclusively via point estimate metrics RMSE R2.<n>We introduce ScoringBench an open benchmark that computes a comprehensive suite of proper scoring rules like CRPS CRLS Interval Score Energy Score weighted CRPS and Brier Score alongside standard point metrics.<n>Results confirm that model rankings depend on the chosen scoring rule and that no single pretraining objective is universally optimal.
  arXiv  Detail & Related papers  (2026-03-31T16:01:56Z)
• Uncertainty Quantification for Regression using Proper Scoring Rules [76.24649098854219]
  We introduce a unified UQ framework for regression based on proper scoring rules, such as CRPS, logarithmic, squared error, and quadratic scores.<n>We derive closed-form expressions for the uncertainty measures under practical parametric assumptions and show how to estimate them using ensembles of models.<n>Our broad evaluation on synthetic and real-world regression datasets provides guidance for selecting reliable UQ measures.
  arXiv  Detail & Related papers  (2025-09-30T17:52:12Z)
• Reweighted Flow Matching via Unbalanced OT for Label-free Long-tailed Generation [9.488297561764211]
  We propose Unbalanced Optimal Transport Reweighted Flow Matching (UOT-RFM) for long-tailed distributions.<n>Our method mitigates majority bias through a principled inverse reweighting strategy.<n>Our model outperforms existing flow matching baselines on long-tailed benchmarks.
  arXiv  Detail & Related papers  (2025-09-30T03:19:52Z)
• TabResFlow: A Normalizing Spline Flow Model for Probabilistic Univariate Tabular Regression [10.521468881948008]
  We introduce TabResFlow, a Normalizing Spline Flow model designed specifically for univariate tabular regression.<n>We evaluate TabResFlow on nine public benchmark datasets, demonstrating that it consistently surpasses existing probabilistic regression models on likelihood scores.<n>We also validate the practical applicability of TabResFlow in a real-world used car price prediction task under selective regression.
  arXiv  Detail & Related papers  (2025-08-23T15:09:02Z)
• How Benchmark Prediction from Fewer Data Misses the Mark [18.693874781163657]
  Benchmark prediction aims to select a small subset of evaluation points and predict overall benchmark performance from that subset.<n>This paper systematically assesses the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks.
  arXiv  Detail & Related papers  (2025-06-09T11:50:41Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models [48.96686419141881]
  We introduce the first formal probabilistic evaluation framework for Large Language Models (LLMs)<n> Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model.<n>Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment.
  arXiv  Detail & Related papers  (2024-10-04T15:44:23Z)
• A Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attributions [60.06461883533697]
  We first identify a set of fidelity criteria that reliable benchmarks for attribution methods are expected to fulfill.<n>We then introduce a Backdoor-based eXplainable AI benchmark (BackX) that adheres to the desired fidelity criteria.<n>Our analysis also offers insights into defending against neural Trojans by utilizing the attributions.
  arXiv  Detail & Related papers  (2024-05-02T13:48:37Z)
• Regression Trees for Fast and Adaptive Prediction Intervals [2.6763498831034043]
  We present a family of methods to calibrate prediction intervals for regression problems with local coverage guarantees. We create a partition by training regression trees and Random Forests on conformity scores. Our proposal is versatile, as it applies to various conformity scores and prediction settings.
  arXiv  Detail & Related papers  (2024-02-12T01:17:09Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression [91.3373131262391]
  Uncertainty is the only certainty there is. Traditionally, the direct regression formulation is considered and the uncertainty is modeled by modifying the output space to a certain family of probabilistic distributions. How to model the uncertainty within the present-day technologies for regression remains an open issue.
  arXiv  Detail & Related papers  (2021-03-25T06:56:09Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.