Deep Learning with Tabular Data: A Self-supervised Approach

• URL: http://arxiv.org/abs/2401.15238v1
• Date: Fri, 26 Jan 2024 23:12:41 GMT
• Title: Deep Learning with Tabular Data: A Self-supervised Approach
• Authors: Tirth Kiranbhai Vyas
• Abstract summary: We have used a self-supervised learning approach in this study. The aim is to find the most effective TabTransformer model representation of categorical and numerical features. The research has presented with a novel approach by creating various variants of TabTransformer model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We have described a novel approach for training tabular data using the TabTransformer model with self-supervised learning. Traditional machine learning models for tabular data, such as GBDT are being widely used though our paper examines the effectiveness of the TabTransformer which is a Transformer based model optimised specifically for tabular data. The TabTransformer captures intricate relationships and dependencies among features in tabular data by leveraging the self-attention mechanism of Transformers. We have used a self-supervised learning approach in this study, where the TabTransformer learns from unlabelled data by creating surrogate supervised tasks, eliminating the need for the labelled data. The aim is to find the most effective TabTransformer model representation of categorical and numerical features. To address the challenges faced during the construction of various input settings into the Transformers. Furthermore, a comparative analysis is also been conducted to examine performance of the TabTransformer model against baseline models such as MLP and supervised TabTransformer. The research has presented with a novel approach by creating various variants of TabTransformer model namely, Binned-TT, Vanilla-MLP-TT, MLP- based-TT which has helped to increase the effective capturing of the underlying relationship between various features of the tabular dataset by constructing optimal inputs. And further we have employed a self-supervised learning approach in the form of a masking-based unsupervised setting for tabular data. The findings shed light on the best way to represent categorical and numerical features, emphasizing the TabTransormer performance when compared to established machine learning models and other self-supervised learning methods.

Related papers

• TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling [28.37672139176765]
  We start by developing TabM -- a simple model based on and our variations of BatchEnembles (an existing technique) In particular, we find that TabM demonstrates the best performance among tabular DL models.
  arXiv  Detail & Related papers  (2024-10-31T17:58:41Z)
• TabDiff: a Multi-Modal Diffusion Model for Tabular Data Generation [91.50296404732902]
  We introduce TabDiff, a joint diffusion framework that models all multi-modal distributions of tabular data in one model. Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data. TabDiff achieves superior average performance over existing competitive baselines, with up to $22.5%$ improvement over the state-of-the-art model on pair-wise column correlation estimations.
  arXiv  Detail & Related papers  (2024-10-27T22:58:47Z)
• A Survey on Deep Tabular Learning [0.0]
  Tabular data presents unique challenges for deep learning due to its heterogeneous nature and lack of spatial structure. This survey reviews the evolution of deep learning models for Tabular data, from early fully connected networks (FCNs) to advanced architectures like TabNet, SAINT, TabTranSELU, and MambaNet.
  arXiv  Detail & Related papers  (2024-10-15T20:08:08Z)
• Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
  The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. We present TP-BERTa, a specifically pre-trained LM for tabular data prediction. A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
  arXiv  Detail & Related papers  (2024-03-04T08:38:56Z)
• TabMT: Generating tabular data with masked transformers [0.0]
  Masked Transformers are incredibly effective as generative models and classifiers. This work contributes to the exploration of transformer-based models in synthetic data generation for diverse application domains.
  arXiv  Detail & Related papers  (2023-12-11T03:28:11Z)
• Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation [67.30538142519067]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM) A deep neural network is then trained to associate these meta-representations with dataset-specific classification confidences. Experiments validate that TabPTM achieves promising performance in new datasets, even under few-shot scenarios.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers [56.00476706550681]
  Table Transformer (TabTransformer) is a state-of-the-art neural network model, while Differential Privacy (DP) is an essential component to ensure data privacy. In this paper, we explore the benefits of combining these two aspects together in the scenario of transfer learning.
  arXiv  Detail & Related papers  (2023-09-12T19:08:26Z)
• Efficient GPT Model Pre-training using Tensor Train Matrix Representation [65.96485282393361]
  Large-scale transformer models feature billions of parameters, leading to difficulties in their deployment and prohibitive training costs from scratch. To reduce the number of parameters in the GPT-2 architecture, we replace the matrices of fully-connected layers with the corresponding Train Matrix(TTM) structure. The resulting GPT-based model stores up to 40% fewer parameters, showing the perplexity comparable to the original model.
  arXiv  Detail & Related papers  (2023-06-05T08:38:25Z)
• XTab: Cross-table Pretraining for Tabular Transformers [29.419276738753968]
  XTab is a framework for cross-table pretraining of tabular transformers on datasets from various domains. We show that XTab consistently boosts the generalizability, learning speed, and performance of multiple tabular transformers. We achieve superior performance than other state-of-the-art tabular deep learning models on various tasks such as regression, binary, and multiclass classification.
  arXiv  Detail & Related papers  (2023-05-10T12:17:52Z)
• Vision Transformers are Robust Learners [65.91359312429147]
  We study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners.
  arXiv  Detail & Related papers  (2021-05-17T02:39:22Z)
• TabTransformer: Tabular Data Modeling Using Contextual Embeddings [23.509063910635692]
  We propose TabTransformer, a novel deep data modeling architecture for supervised and semi-supervised learning. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. For the semi-supervised setting, we develop an unsupervised pre-training procedure to learn data-driven contextual embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-12-11T23:31:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.