Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap

• URL: http://arxiv.org/abs/2605.18696v1
• Date: Mon, 18 May 2026 17:32:57 GMT
• Title: Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap
• Abstract summary: No single Tabular foundation models (TFMs) wins on every dataset.<n>We benchmark six ensemble strategies over six TFMs on 153 classification tasks.<n>The best ensemble, two-level cascade stacking, buys $+0.18%$ accuracy over the strongest single TFM at $253times$ the compute.
• Score: 3.478921293603811
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tabular foundation models (TFMs) now match or beat tuned gradient-boosted trees on a growing fraction of tabular tasks, but no single TFM wins on every dataset. Ensembling is the go to fix here, and it works less well than expected. Six modern TFMs form a near-redundant pool: their mean pairwise Q-statistic is $0.961$, close enough to $1$ that any convex combination is bounded above. We benchmark six ensemble strategies over six TFMs on 153 OpenML classification tasks. The best ensemble, two-level cascade stacking, buys $+0.18\%$ accuracy over the strongest single TFM at $253\times$ the compute. A Friedman and Nemenyi analysis places three ensembles and the best base TFM in a single equivalence group; three other ensembles are significantly \emph{worse} than the best base. Stacking with a logistic-regression meta-learner is the most striking case: competitive accuracy and ROC-AUC, the worst log-loss rank among the ensembles. The meta-learner improves accuracy by sharpening class boundaries, which destroys calibration. We recommend greedy selection as the practical default.

Related papers

• Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning [64.49364696628294]
  We introduce textbfOckhamareto, a single-shot GRPO framework for unit-test generation and selection.<n>Ockhamareto has two principal components: (i)a emphPareto-gated Bonus that rewards only rollouts non-dominated in(mutation, $-$#tests) space, and (ii)emphToken-level Segment Credit, which attributes each test's marginal mutation kills back to the tokens of its unit-test block.
  arXiv  Detail & Related papers  (2026-08-25T12:20:02Z)
• Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Lift [0.0]
  This paper provides an experimentally verified formal law for calculating the uplift that diversity of thought provides in Large Language Model ensembles.<n>We test the law on 767,520 inferences from ten open-weight models over two graduate-level science benchmarks.
  arXiv  Detail & Related papers  (2026-07-19T19:01:06Z)
• Per-Group Error, Not Total MSE: Fine-Tuning Vision-Language-Action Models for 11-DoF Mobile Manipulation [4.798596809049421]
  Per-group analysis exposes two patterns: in SmolVLA, the mobile base converges slowest and limits overall performance.<n>In expert-only fine-tuning of $_0.5$ (training only the action head, backbone frozen), total MSE drops below the baseline but arm accuracy degrades.<n>We conclude that per-group error is a more reliable signal than total MSE for checkpoint selection on robots with heterogeneous action spaces.
  arXiv  Detail & Related papers  (2026-05-29T18:36:08Z)
• LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning [0.2999888908665658]
  We propose LoMETab, a rank-$r$ generalization of multiplicative implicit ensembles.<n>LoMETab lifts the rank-1 BatchEnsemble/TabM modulation to a rank-$r$ identity-residual Hadamard family.<n> Empirically, we show that this added capacity manifests as measurable predictive diversity after training.
  arXiv  Detail & Related papers  (2026-05-14T04:47:16Z)
• What Do EEG Foundation Models Capture from Human Brain Signals? [64.48249643001402]
  Modern EEG foundation models learn directly from raw signals via self-supervised pretraining.<n>We decompose into three sub-questions: emphwhat does the model learn, emphwhat does the model use, and emphhow much can be explained<n>The audit covers three foundation models (CSBrain, CBraMod, LaBraM), five clinical tasks (MDD, Stress, ISRUC-Sleep, TUSL, Siena), and a 6-family 63-feature lexicon.
  arXiv  Detail & Related papers  (2026-05-12T01:57:53Z)
• A Boundary-Metric Evaluation Protocol for Whiteboard Stroke Segmentation Under Extreme Imbalance [0.0]
  Boundary-aware metrics and a thin-subset equity analysis change how loss functions rank and exposes hidden trade-offs.<n>Five losses are trained three times each on a DeepLabV3-MobileNetV3 model and evaluated on 12 held-out images.<n>Overlap-based losses improve F1 by more than 20 points over cross-entropy.
  arXiv  Detail & Related papers  (2026-02-26T05:16:18Z)
• Trade-off in Estimating the Number of Byzantine Clients in Federated Learning [51.015234193544636]
  Federated learning is vulnerable to Byzantine clients that can send any erroneous signals.<n>We show that underestimation ($hatfge f$) can lead to arbitrarily poor performance for both aggregators and federated learning.<n>All these optimal bounds are proportional to $hatf/(n-f-hatf)$ with $n$ clients, which monotonically increases with larger $hatf$.
  arXiv  Detail & Related papers  (2025-10-06T02:01:56Z)
• RAMP: Boosting Adversarial Robustness Against Multiple $l_p$ Perturbations for Universal Robustness [4.188296977882316]
  We propose a novel training framework textbfRAMP, to boost the robustness against multiple $l_p$ perturbations. For training from scratch, textbfRAMP achieves a union accuracy of $44.6%$ and good clean accuracy of $81.2%$ on ResNet-18 against AutoAttack on CIFAR-10.
  arXiv  Detail & Related papers  (2024-02-09T23:29:54Z)
• Solving a Class of Non-Convex Minimax Optimization in Federated Learning [84.98927714326908]
  The minimax problems arise throughout machine learning applications, ranging from machine learning training to large-scale learning. We propose a class of algorithms for non minimax problems (emphi) that reduce complexity to $varepsilon-6)$. We prove that FedSGDA-M has the best sample complexity of $O(kappa2-3)$ and the best-known communication of $O(kappa2-3)$.
  arXiv  Detail & Related papers  (2023-10-05T15:48:41Z)
• Contextual Combinatorial Bandits with Probabilistically Triggered Arms [55.9237004478033]
  We study contextual bandits with probabilistically triggered arms (C$2$MAB-T) under a variety of smoothness conditions. Under the triggering modulated (TPM) condition, we devise the C$2$-UC-T algorithm and derive a regret bound $tildeO(dsqrtT)$.
  arXiv  Detail & Related papers  (2023-03-30T02:51:00Z)
• Rebalanced Siamese Contrastive Mining for Long-Tailed Recognition [120.80038161330623]
  We show that supervised contrastive learning suffers a dual class-imbalance problem at both the original batch and Siamese batch levels. We propose supervised hard positive and negative pairs mining to pick up informative pairs for contrastive computation and improve representation learning.
  arXiv  Detail & Related papers  (2022-03-22T07:30:38Z)
• Modality-Aware Triplet Hard Mining for Zero-shot Sketch-Based Image Retrieval [51.42470171051007]
  This paper tackles the Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) problem from the viewpoint of cross-modality metric learning. By combining two fundamental learning approaches in DML, e.g., classification training and pairwise training, we set up a strong baseline for ZS-SBIR. We show that Modality-Aware Triplet Hard Mining (MATHM) enhances the baseline with three types of pairwise learning.
  arXiv  Detail & Related papers  (2021-12-15T08:36:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.