A Mechanistic Study of Tabular Foundation Models

• URL: http://arxiv.org/abs/2605.21288v1
• Date: Wed, 20 May 2026 15:23:16 GMT
• Title: A Mechanistic Study of Tabular Foundation Models
• Abstract summary: Tabular foundation models converge in accuracy across a range of classification and regression tasks.<n>This raises questions a leaderboard cannot answer.<n>We characterize all three: (i) whether the models execute the same in-context algorithm, (ii) where row, column, and class-permutation invariances originate, and (iii) how robust they are under perturbations engineered against the inferred mechanism.
• Score: 12.26738662548555
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular foundation models with different architectures converge in accuracy across a range of classification and regression tasks. This raises questions a leaderboard cannot answer: (i) whether the models execute the same in-context algorithm, (ii) where row, column, and class-permutation invariances originate, and (iii) how robust they are under perturbations engineered against the inferred mechanism. We characterize all three. The model families realize qualitatively distinct similarity-based readouts: from an attention-weighted vote over context labels to a class-conditional mean readout, each confirmed by causal intervention. We find that the representation collapse highlighted in prior work is not a practical concern for them. Each model's permutation invariances trace to specific positional parameters whose removal preserves accuracy and makes approximate invariance exact. Perturbations engineered against each readout reproduce predicted failure modes; hub and rank attacks isolate them from refit baselines. Together these results give a mechanistic account of contemporary tabular foundation models and identify which inductive biases govern both their accuracy and characteristic failures.

Related papers

• Evaluating Local Explainability Metrics for Machine Learning Models on Tabular Data [1.423280626666929]
  An explanation can appear plausible to humans but fail to capture the internal reasoning of a model.<n>This paper studies the trustworthiness of local explainability techniques when applied to complex tabular classification tasks.
  arXiv  Detail & Related papers  (2026-05-26T19:38:10Z)
• LLMs Show No Signs Of Individuated Metacognition [0.023227405857540805]
  We decompose binary confidence judgements from 20 Large Language Models.<n>We ask whether two models that differ in confidence also differ in performance.<n>We find no evidence for significant verbalised individuated metacognition in any tested domain.
  arXiv  Detail & Related papers  (2026-05-22T23:54:33Z)
• Learning Causal Orderings for In-Context Tabular Prediction [37.169402067005144]
  In this paper, we study how to infer and enforce causal structure in the form of topological variable orderings.<n>Our model TabOrder uses causal order-constrained attention, basing predictions only on features that precede a target under a learned causal order.<n>Similar to causal discovery methods, TabOrder learns the optimal variable ordering in an unsupervised manner through a likelihood-based objective.
  arXiv  Detail & Related papers  (2026-05-21T11:22:56Z)
• Learn to Rank: Visual Attribution by Learning Importance Ranking [58.69028273772474]
  We propose a new approach to generate visual attribution maps for computer vision models.<n>Our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement.<n>Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations.
  arXiv  Detail & Related papers  (2026-04-07T12:53:22Z)
• Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution [0.0]
  We find that models with identical predictive behavior can produce substantially different feature attributions.<n>We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery.<n>Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.
  arXiv  Detail & Related papers  (2026-03-16T18:55:50Z)
• Indiscriminate Disruption of Conditional Inference on Multivariate Gaussians [60.22542847840578]
  Despite advances in adversarial machine learning, inference for Gaussian models in the presence of an adversary is notably understudied. We consider a self-interested attacker who wishes to disrupt a decisionmaker's conditional inference and subsequent actions by corrupting a set of evidentiary variables. To avoid detection, the attacker also desires the attack to appear plausible wherein plausibility is determined by the density of the corrupted evidence.
  arXiv  Detail & Related papers  (2024-11-21T17:46:55Z)
• Attuned to Change: Causal Fine-Tuning under Latent-Confounded Shifts [32.989526411946606]
  Adapting to latent-confounded shifts remains a core challenge in modern AI.<n>One practical failure mode arises when fine-tuning pre-trained foundation models on confounded data.<n>We frame causal fine-tuning as an identification problem and pose an explicit causal model that decomposes inputs into low-level spurious features.
  arXiv  Detail & Related papers  (2024-10-18T11:06:23Z)
• From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication [19.336940758147442]
  It has been observed that representations learned by distinct neural networks conceal structural similarities when the models are trained under similar inductive biases. We introduce a versatile method to directly incorporate a set of invariances into the representations, constructing a product space of invariant components on top of the latent representations. We validate our solution on classification and reconstruction tasks, observing consistent latent similarity and downstream performance improvements in a zero-shot stitching setting.
  arXiv  Detail & Related papers  (2023-10-02T13:55:38Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Resolving label uncertainty with implicit posterior models [71.62113762278963]
  We propose a method for jointly inferring labels across a collection of data samples. By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs.
  arXiv  Detail & Related papers  (2022-02-28T18:09:44Z)
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests [87.60900567941428]
  A spurious correlation' is the dependence of a model on some aspect of the input data that an analyst thinks shouldn't matter. In machine learning, these have a know-it-when-you-see-it character. We study stress testing using the tools of causal inference.
  arXiv  Detail & Related papers  (2021-05-31T14:39:38Z)
• Tractable Inference in Credal Sentential Decision Diagrams [116.6516175350871]
  Probabilistic sentential decision diagrams are logic circuits where the inputs of disjunctive gates are annotated by probability values. We develop the credal sentential decision diagrams, a generalisation of their probabilistic counterpart that allows for replacing the local probabilities with credal sets of mass functions. For a first empirical validation, we consider a simple application based on noisy seven-segment display images.
  arXiv  Detail & Related papers  (2020-08-19T16:04:34Z)
• Reachable Sets of Classifiers and Regression Models: (Non-)Robustness Analysis and Robust Training [1.0878040851638]
  We analyze and enhance robustness properties of both classifiers and regression models. Specifically, we verify (non-)robustness, propose a robust training procedure, and show that our approach outperforms adversarial attacks. Second, we provide techniques to distinguish between reliable and non-reliable predictions for unlabeled inputs, to quantify the influence of each feature on a prediction, and compute a feature ranking.
  arXiv  Detail & Related papers  (2020-07-28T10:58:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.