Arithmetic Feature Interaction Is Necessary for Deep Tabular Learning

• URL: http://arxiv.org/abs/2402.02334v2
• Date: Tue, 19 Mar 2024 11:55:14 GMT
• Title: Arithmetic Feature Interaction Is Necessary for Deep Tabular Learning
• Authors: Yi Cheng, Renjun Hu, Haochao Ying, Xing Shi, Jian Wu, Wei Lin,
• Abstract summary: We create a synthetic dataset with a mild feature interaction assumption. We examine a modified transformer architecture enabling arithmetical feature interactions, referred to as AMFormer.
• Score: 17.38478179159257
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Until recently, the question of the effective inductive bias of deep models on tabular data has remained unanswered. This paper investigates the hypothesis that arithmetic feature interaction is necessary for deep tabular learning. To test this point, we create a synthetic tabular dataset with a mild feature interaction assumption and examine a modified transformer architecture enabling arithmetical feature interactions, referred to as AMFormer. Results show that AMFormer outperforms strong counterparts in fine-grained tabular data modeling, data efficiency in training, and generalization. This is attributed to its parallel additive and multiplicative attention operators and prompt-based optimization, which facilitate the separation of tabular samples in an extended space with arithmetically-engineered features. Our extensive experiments on real-world data also validate the consistent effectiveness, efficiency, and rationale of AMFormer, suggesting it has established a strong inductive bias for deep learning on tabular data. Code is available at https://github.com/aigc-apps/AMFormer.

Related papers

• APAR: Modeling Irregular Target Functions in Tabular Regression via Arithmetic-Aware Pre-Training and Adaptive-Regularized Fine-Tuning [12.35924469567586]
  We propose a novel Arithmetic-Aware Pre-training and Adaptive-Regularized Fine-tuning framework (APAR) In the pre-training phase, APAR introduces an arithmetic-aware pretext objective to capture intricate sample-wise relationships from the perspective of continuous labels. In the fine-tuning phase, a consistency-based adaptive regularization technique is proposed to self-learn appropriate data augmentation.
  arXiv  Detail & Related papers  (2024-12-14T19:33:21Z)
• Tabular data generation with tensor contraction layers and transformers [0.35998666903987897]
  We investigate the potential of using embedding representations on data generation, utilizing tensor contraction layers and transformers. Our empirical study, conducted across multiple datasets from the OpenML CC18 suite, compares models over density estimation and Machine Learning efficiency metrics. The main takeaway from our results is that leveraging embedding representations with the help of tensor contraction layers improves density estimation metrics, albeit maintaining competitive performance in terms of machine learning efficiency.
  arXiv  Detail & Related papers  (2024-12-06T19:34:13Z)
• TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation [91.50296404732902]
  We introduce TabDiff, a joint diffusion framework that models all mixed-type distributions of tabular data in one model. Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data. TabDiff achieves superior average performance over existing competitive baselines, with up to $22.5%$ improvement over the state-of-the-art model on pair-wise column correlation estimations.
  arXiv  Detail & Related papers  (2024-10-27T22:58:47Z)
• A Survey on Deep Tabular Learning [0.0]
  Tabular data presents unique challenges for deep learning due to its heterogeneous nature and lack of spatial structure. This survey reviews the evolution of deep learning models for Tabular data, from early fully connected networks (FCNs) to advanced architectures like TabNet, SAINT, TabTranSELU, and MambaNet.
  arXiv  Detail & Related papers  (2024-10-15T20:08:08Z)
• Distributionally robust self-supervised learning for tabular data [2.942619386779508]
  Learning robust representation in presence of error slices is challenging, due to high cardinality features and the complexity of constructing error sets. Traditional robust representation learning methods are largely focused on improving worst group performance in supervised setting in computer vision. Our approach utilizes an encoder-decoder model trained with Masked Language Modeling (MLM) loss to learn robust latent representations.
  arXiv  Detail & Related papers  (2024-10-11T04:23:56Z)
• Discovering symbolic expressions with parallelized tree search [59.92040079807524]
  Symbolic regression plays a crucial role in scientific research thanks to its capability of discovering concise and interpretable mathematical expressions from data. Existing algorithms have faced a critical bottleneck of accuracy and efficiency over a decade when handling problems of complexity. We introduce a parallelized tree search (PTS) model to efficiently distill generic mathematical expressions from limited data.
  arXiv  Detail & Related papers  (2024-07-05T10:41:15Z)
• Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
  The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. We present TP-BERTa, a specifically pre-trained LM for tabular data prediction. A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
  arXiv  Detail & Related papers  (2024-03-04T08:38:56Z)
• Automatic Data Augmentation via Invariance-Constrained Learning [94.27081585149836]
  Underlying data structures are often exploited to improve the solution of learning tasks. Data augmentation induces these symmetries during training by applying multiple transformations to the input data. This work tackles these issues by automatically adapting the data augmentation while solving the learning task.
  arXiv  Detail & Related papers  (2022-09-29T18:11:01Z)
• Why do tree-based models still outperform deep learning on tabular data? [0.0]
  We show that tree-based models remain state-of-the-art on medium-sized data. We conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs)
  arXiv  Detail & Related papers  (2022-07-18T08:36:08Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Relation-Guided Representation Learning [53.60351496449232]
  We propose a new representation learning method that explicitly models and leverages sample relations. Our framework well preserves the relations between samples. By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
  arXiv  Detail & Related papers  (2020-07-11T10:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.