LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning

• URL: http://arxiv.org/abs/2605.14365v1
• Date: Thu, 14 May 2026 04:47:16 GMT
• Title: LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning
• Abstract summary: We propose LoMETab, a rank-$r$ generalization of multiplicative implicit ensembles.<n>LoMETab lifts the rank-1 BatchEnsemble/TabM modulation to a rank-$r$ identity-residual Hadamard family.<n> Empirically, we show that this added capacity manifests as measurable predictive diversity after training.
• Score: 0.2999888908665658
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent tabular learning benchmarks increasingly show a tight performance cluster rather than a clear hierarchy among leading methods, spanning gradient boosted decision trees, attention-based architectures, and implicit ensembles such as TabM. As benchmark gains plateau, a complementary goal is to understand and control the mechanisms that make simple neural tabular models competitive. We propose LoMETab, a rank-$r$ generalization of multiplicative implicit ensembles. LoMETab lifts the rank-1 BatchEnsemble/TabM modulation to a rank-$r$ identity-residual Hadamard family by parameterizing each member weight as $W_k = W \odot (1 + A_kB_k^\top)$, where $W$ is shared and $(A_k, B_k)$ are member-specific low-rank factors. This exposes two practical diversity-control axes: the adapter rank $r$ and the initialization scale $σ_{\mathrm{init}}$, and we prove that for $r \ge 2$ this generalization strictly enlarges BatchEnsemble's hypothesis class. Empirically, we show that this added capacity manifests as measurable predictive diversity after training: on representative classification datasets, LoMETab sustains higher pairwise KL than an additive low-rank ablation, and $(r, σ_{\mathrm{init}})$ provides broad control over pairwise KL, varying by up to several orders of magnitude across configurations. The induced diversity is reflected in task-appropriate output-level measures: argmax disagreement for classification and ambiguity for regression, indicating that the control extends beyond pairwise KL to decision- and output-level member variation. Finally, experiments sweeping over adapter rank $r$ and initialization scale $σ_{\mathrm{init}}$ reveal that predictive performance is dataset-dependent over the $(r, σ_{\mathrm{init}})$ grid, supporting LoMETab as a controllable family of implicit ensembles rather than a fixed rank-1 construction.

Related papers

• Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate [0.0]
  Mixed-precision quantization must decide which parts of a model to keep at higher precision.<n>We show that the loss from quantizing a set of layers can be reconstructed from per-layer or pairwise sensitivities measured in isolation.<n>We propose the coverage model $f(S)=cbigl (1-prod_iin S (1-a_i)bigr)$, which reproduces the measured variance profile of $f$ to within a few percent from its $L$ fitted break-rates.
  arXiv  Detail & Related papers  (2026-07-14T02:08:33Z)
• A Complexity Measure for Active Learning in Multi-group Mean Estimation [8.550300650352732]
  We study a emphmax-risk objective for active learning in a multi-group mean estimation $d$-armed bandits.<n>A learner adaptively allocates a budget of $T$ samples across $d$ groups to minimize the worst-case uncertainty index.<n>For smooth classes, the $mathrmVLC$ is a reparametrization of a variance--Fisher information, with closed-form values for common families.
  arXiv  Detail & Related papers  (2026-06-12T17:54:26Z)
• The Sample Complexity of Multiclass and Sparse Contextual Bandits [106.74652380822778]
  We aim to identify an approximately optimal policy from a given class based on bandit feedback.<n>Motivated by bandit multiclass classification with zero-one rewards, we focus on the emph$s$-sparse setting.<n>We show that, with $s$-sparse rewards, the induced model class admits a sharp DEC bound that scales with $s$ and directly yields the optimal rate.
  arXiv  Detail & Related papers  (2026-05-28T09:12:20Z)
• Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks [8.250374560598493]
  We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries.<n>We show that a student trained with textscAdam near the effective width $k_c$ achieves these optimal scaling laws.
  arXiv  Detail & Related papers  (2026-05-11T11:39:03Z)
• Near-Optimal Regret for KL-Regularized Multi-Armed Bandits [54.77408659142336]
  We study the statistical efficiency of online learning with respect to KL-regularized objectives.<n>We show that the KL-regularized regret for MABs is $$-independent and scales as $tilde(sqrtKT)$.
  arXiv  Detail & Related papers  (2026-03-02T18:17:33Z)
• Leave-One-Out Prediction for General Hypothesis Classes [9.855978207725549]
  We introduce Median of Level-Set Aggregation (MLSA), a general aggregation procedure based on empirical-risk level sets around the ERM.<n>For arbitrary fixed datasets and losses satisfying a mild monotonicity condition, we establish a multiplicative oracle inequality for the LOO error of the form [ LOO_S(hath) ;le; C cdot frac1n min_hin H L_S(h) ;+; fracComp(S,
  arXiv  Detail & Related papers  (2026-03-02T16:27:44Z)
• Do Schwartz Higher-Order Values Help Sentence-Level Human Value Detection? When Hard Gating Hurts [3.0339361014383397]
  We study whether Schwartz higher-order (HO) categories provide usable structure under compute-frugal budget (single 8 GB GPU)<n>We compare direct supervised transformers, (ii) HO$rightarrow$values pipelines that enforce the hierarchy with hard masks, and (iii) Presence$rightarrow$HO$rightarrow$values cascades, alongside low-cost add-ons (lexica, short context, topics)<n>Overall, HO structure is useful descriptively, but enforcing it with hard gates hurts sentence-level value detection; robust improvements come from calibration and lightweight ensembling.
  arXiv  Detail & Related papers  (2026-01-31T21:50:35Z)
• Multiple-play Stochastic Bandits with Prioritized Arm Capacity Sharing [52.124267908936396]
  The model is composed of $M$ arms and $K$ plays.<n>Each arm has a number of capacities, and each unit of capacity is associated with a reward function.<n>When multiple plays compete for the arm capacity, the arm capacity is allocated in a larger priority weight first manner.
  arXiv  Detail & Related papers  (2025-12-25T11:19:09Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Fine-Grained Gap-Dependent Bounds for Tabular MDPs via Adaptive Multi-Step Bootstrap [84.66885506098724]
  This paper presents a new model-free algorithm for episodic finite-horizon Markov Decision Processes (MDP), Adaptive Multi-step Bootstrap (AMB) We show AMB achieves a gap-dependent regret bound that only scales with the sum of the inverse of the sub-optimality gaps. We also show AMB suffers an additional $frac|Z_mul|Delta_min$ regret, where $Z_mul$ is the set of state-action pairs $(s,a)$'s satisfying $a$ is a non-unique optimal action for
  arXiv  Detail & Related papers  (2021-02-09T07:46:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.