Revisiting Metafeatures to Explain Model Differences on Tabular Data

• URL: http://arxiv.org/abs/2605.28418v2
• Date: Thu, 28 May 2026 09:14:40 GMT
• Title: Revisiting Metafeatures to Explain Model Differences on Tabular Data
• Abstract summary: Using the TabArena benchmark results, we analyze dataset-level performance gaps and relate them to model-agnostic dataset descriptors.<n>We find that for neural network vs. tree gaps, no meta-feature survives false discovery control, and one association is robust but does not generalize when tested in leave-one-dataset-out prediction.
• Score: 12.445205798094412
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the rise of tabular foundation models alongside traditional models still performing well on many tasks, choosing the right model for a tabular dataset remains difficult. We investigate whether dataset meta-features can explain performance gaps between model families on tabular prediction tasks. Using the TabArena benchmark results, we analyze dataset-level performance gaps and relate them to model-agnostic dataset descriptors. After strict statistical tests with false discovery control, we find that (1) for neural network vs. tree gaps, no meta-feature survives false discovery control, (2) for non-foundation vs. foundation model gaps, one association is robust but does not generalize when tested in leave-one-dataset-out prediction, and (3) for TabICLv2 vs. TabPFN-2.6, one robust association also improves held-out prediction. Furthermore, we conduct a leave-one-dataset-out analysis and find that meta-feature predictors fail to improve meaningfully over a simple baseline. Overall, our results show the heterogeneity of tabular datasets and that global meta-feature approaches are not robust enough to offer explanations on the 51 TabArena datasets.

Related papers

• Beyond IID: How General Are Tabular Foundation Models, Really? [76.03669797389378]
  Research communities are increasingly evaluating foundation models on diverse datasets and tasks.<n>Standard benchmarks are mostly defined for tasks where tabular foundation models already excel.<n>We introduce BeyondArena, the first unified holistic benchmark for tabular data.
  arXiv  Detail & Related papers  (2026-06-29T14:55:52Z)
• Tabular Foundation Model for Generative Modelling [34.3599321018728]
  Generative modelling requires robust, holistic representation learning for a given data modality, rather than optimisation for a supervised prediction target alone.<n>Existing tabular foundation generators have not yet consistently matched strong dataset-specific generators in synthetic data quality.<n>We introduce TabFORGE, built on pretrained textbfTabular textbfFOundational textbfRepresentations for textbfGEneration.
  arXiv  Detail & Related papers  (2026-05-10T08:52:28Z)
• Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models [7.124188498356204]
  Tabular foundation models are pre-trained on one of three classes of corpus: curated datasets drawn from benchmark repositories, tables harvested at scale from the web, or synthetic tables sampled from a parametric generative prior.<n>In this work we take three canonical, archetypal datasets used to train tabular foundation models.<n>We characterise each corpus using aggregate features over whole tables, columns and correlations, and compare them using discriminator AUCs and k-NN coverage metrics.<n>We find that the TabICL synthetic prior occupies a narrow region of the space of real tables, that this mismatch cannot be closed by optimising prior
  arXiv  Detail & Related papers  (2026-05-07T14:29:31Z)
• Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models [10.044939709126007]
  Generalized additive models (GAMs) often rely on procedures to select interactions, potentially missing higher-order or context-dependent effects.<n>To meet this challenge, we propose TabDistill, a method that leverages tabular foundation models and post-hoc distillation methods.<n>Our results suggest that TabDistill can serve as effective, data-driven guides for interaction discovery, bridging high-capacity models and interpretable additive frameworks.
  arXiv  Detail & Related papers  (2026-04-14T22:37:11Z)
• OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale [82.32822441887886]
  We introduce OmniTabBench, the largest benchmark to date, comprising 3030 datasets spanning diverse tasks.<n>We conduct an unprecedented large-scale empirical evaluation of state-of-the-art models from all model families on OmniTabBench, confirming the absence of a dominant winner.
  arXiv  Detail & Related papers  (2026-04-08T08:31:43Z)
• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard [27.224577475861214]
  Tabular data, a fundamental data format in machine learning, is predominantly utilized in competitions and real-world applications.<n>We propose an in-context ensemble framework for tabular prediction that leverages large language models.<n>Our method constructs a context around each test instance using its nearest neighbors and the predictions from a pool of external models.
  arXiv  Detail & Related papers  (2025-05-19T17:52:58Z)
• A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities [51.08999772842298]
  Tabular Prior-data Fitted Network v2 (TabPFN v2) achieves unprecedented in-context learning performance across diverse downstream datasets.<n>We show that TabPFN v2 can infer attribute relationships even when provided with randomized attribute token inputs.<n>We demonstrate that TabPFN v2's limitations can be addressed through a test-time divide-and-context strategy.
  arXiv  Detail & Related papers  (2025-02-24T17:38:42Z)
• Fully Test-time Adaptation for Tabular Data [48.67303250592189]
  We propose the Fully Test-time Adaptation for Tabular data, which enables FTTA methods to robustly optimize the label distribution of predictions.<n>We conduct comprehensive experiments on six benchmark datasets, which are evaluated using three metrics.
  arXiv  Detail & Related papers  (2024-12-14T15:49:53Z)
• Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective [71.45945607871715]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM)<n>The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels.<n>Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinement [44.693325083735424]
  Tabular data prediction has been employed in medical applications such as patient health risk prediction. Previous predictors are often trained on manually curated small datasets.
  arXiv  Detail & Related papers  (2023-05-20T03:37:09Z)
• REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers [0.0]
  We introduce REaLTabFormer (Realistic and Tabular Transformer), a synthetic data generation model. It first creates a parent table using an autoregressive GPT-2 model, then generates the relational dataset conditioned on the parent table using a sequence-to-sequence model. Experiments using real-world datasets show that REaLTabFormer captures the relational structure better than a model baseline.
  arXiv  Detail & Related papers  (2023-02-04T00:32:50Z)
• Towards Robustifying NLI Models Against Lexical Dataset Biases [94.79704960296108]
  This paper explores both data-level and model-level debiasing methods to robustify models against lexical dataset biases. First, we debias the dataset through data augmentation and enhancement, but show that the model bias cannot be fully removed via this method. The second approach employs a bag-of-words sub-model to capture the features that are likely to exploit the bias and prevents the original model from learning these biased features.
  arXiv  Detail & Related papers  (2020-05-10T17:56:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.