Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models

• URL: http://arxiv.org/abs/2604.13332v1
• Date: Tue, 14 Apr 2026 22:37:11 GMT
• Title: Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models
• Abstract summary: Generalized additive models (GAMs) often rely on procedures to select interactions, potentially missing higher-order or context-dependent effects.<n>To meet this challenge, we propose TabDistill, a method that leverages tabular foundation models and post-hoc distillation methods.<n>Our results suggest that TabDistill can serve as effective, data-driven guides for interaction discovery, bridging high-capacity models and interpretable additive frameworks.
• Score: 10.044939709126007
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Identifying meaningful feature interactions is a central challenge in building accurate and interpretable models for tabular data. Generalized additive models (GAMs) have shown great success at modeling tabular data, but often rely on heuristic procedures to select interactions, potentially missing higher-order or context-dependent effects. To meet this challenge, we propose TabDistill, a method that leverages tabular foundation models and post-hoc distillation methods. Our key intuition is that tabular foundation models implicitly learn rich, adaptive feature dependencies through large-scale representation learning. Given a dataset, TabDistill first fits a tabular foundation model to the dataset, and then applies a post-hoc interaction attribution method to extract salient feature interactions from it. We evaluate these interactions by then using them as terms in a GAM. Across tasks, we find that interactions identified by TabDistill lead to consistent improvements in downstream GAMs' predictive performance. Our results suggest that tabular foundation models can serve as effective, data-driven guides for interaction discovery, bridging high-capacity models and interpretable additive frameworks.

Related papers

• TS2TabPFN: Time Series Classification and Extrinsic Regression through Feature Extraction and a Tabular Foundation Model [0.7913770737379631]
  This paper proposes TS2TabPFN, a framework that integrates explicit feature extraction with TabPFN 2.5.<n>We show that TS2TabPFN significantly outperforms state-of-the-art models in TSER tasks with statistical significance.<n>Results suggest that combining foundation models with structured features overcomes single-paradigm limitations.
  arXiv  Detail & Related papers  (2026-08-04T19:32:18Z)
• Tabular Foundation Model for Generative Modelling [34.3599321018728]
  Generative modelling requires robust, holistic representation learning for a given data modality, rather than optimisation for a supervised prediction target alone.<n>Existing tabular foundation generators have not yet consistently matched strong dataset-specific generators in synthetic data quality.<n>We introduce TabFORGE, built on pretrained textbfTabular textbfFOundational textbfRepresentations for textbfGEneration.
  arXiv  Detail & Related papers  (2026-05-10T08:52:28Z)
• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation [16.907006955584343]
  Diffusion models have been the predominant generative model for data generation.<n>We present TabRep, a training architecture trained with a unified continuous representation.<n>Our results showcase that TabRep achieves superior performance across a broad suite of evaluations.
  arXiv  Detail & Related papers  (2025-04-07T07:44:27Z)
• MixRec: Heterogeneous Graph Collaborative Filtering [21.96510707666373]
  We present a graph collaborative filtering model MixRec to disentangling users' multi-behavior interaction patterns.<n>Our model achieves this by incorporating intent disentanglement and multi-behavior modeling.<n>We also introduce a novel contrastive learning paradigm that adaptively explores the advantages of self-supervised data augmentation.
  arXiv  Detail & Related papers  (2024-12-18T13:12:36Z)
• A Collaborative Ensemble Framework for CTR Prediction [73.59868761656317]
  We propose a novel framework, Collaborative Ensemble Training Network (CETNet), to leverage multiple distinct models. Unlike naive model scaling, our approach emphasizes diversity and collaboration through collaborative learning. We validate our framework on three public datasets and a large-scale industrial dataset from Meta.
  arXiv  Detail & Related papers  (2024-11-20T20:38:56Z)
• TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation [91.50296404732902]
  We introduce TabDiff, a joint diffusion framework that models all mixed-type distributions of tabular data in one model.<n>Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data.<n>TabDiff achieves superior average performance over existing competitive baselines, with up to $22.5%$ improvement over the state-of-the-art model on pair-wise column correlation estimations.
  arXiv  Detail & Related papers  (2024-10-27T22:58:47Z)
• A Survey on Deep Tabular Learning [0.0]
  Tabular data presents unique challenges for deep learning due to its heterogeneous nature and lack of spatial structure. This survey reviews the evolution of deep learning models for Tabular data, from early fully connected networks (FCNs) to advanced architectures like TabNet, SAINT, TabTranSELU, and MambaNet.
  arXiv  Detail & Related papers  (2024-10-15T20:08:08Z)
• A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gap [50.079224604394]
  We present a novel model-agnostic framework called textbfContext-textbfEnhanced textbfFeature textbfAment (CEFA) CEFA consists of a feature alignment module and a context enhancement module. Our method can serve as a plug-and-play module to improve the detection performance of HOI models on rare categories.
  arXiv  Detail & Related papers  (2024-07-31T08:42:48Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• Variable Importance Matching for Causal Inference [73.25504313552516]
  We describe a general framework called Model-to-Match that achieves these goals. Model-to-Match uses variable importance measurements to construct a distance metric. We operationalize the Model-to-Match framework with LASSO.
  arXiv  Detail & Related papers  (2023-02-23T00:43:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.