Mambular: A Sequential Model for Tabular Deep Learning

• URL: http://arxiv.org/abs/2408.06291v1
• Date: Mon, 12 Aug 2024 16:57:57 GMT
• Title: Mambular: A Sequential Model for Tabular Deep Learning
• Authors: Anton Frederik Thielmann, Manish Kumar, Christoph Weisser, Arik Reuter, Benjamin Säfken, Soheila Samiee,
• Abstract summary: We introduce Mambular, an adaptation of the Mamba architecture optimized for tabular data. We benchmark Mambular against state-of-the-art models, including neural networks and tree-based methods. Our analysis shows that interpreting features as a sequence and passing them through Mamba layers results in surprisingly performant models.
• Score: 0.7184556517162347
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The analysis of tabular data has traditionally been dominated by gradient-boosted decision trees (GBDTs), known for their proficiency with mixed categorical and numerical features. However, recent deep learning innovations are challenging this dominance. We introduce Mambular, an adaptation of the Mamba architecture optimized for tabular data. We extensively benchmark Mambular against state-of-the-art models, including neural networks and tree-based methods, and demonstrate its competitive performance across diverse datasets. Additionally, we explore various adaptations of Mambular to understand its effectiveness for tabular data. We investigate different pooling strategies, feature interaction mechanisms, and bi-directional processing. Our analysis shows that interpreting features as a sequence and passing them through Mamba layers results in surprisingly performant models. The results highlight Mambulars potential as a versatile and powerful architecture for tabular data analysis, expanding the scope of deep learning applications in this domain. The source code is available at https://github.com/basf/mamba-tabular.

Related papers

• Escaping the Forest: Sparse Interpretable Neural Networks for Tabular Data [0.0]
  We show that our models, Sparse TABular NET or sTAB-Net with attention mechanisms, are more effective than tree-based models. They achieve better performance than post-hoc methods like SHAP.
  arXiv  Detail & Related papers  (2024-10-23T10:50:07Z)
• Modern Neighborhood Components Analysis: A Deep Tabular Baseline Two Decades Later [59.88557193062348]
  We revisit the classic Neighborhood Component Analysis (NCA), designed to learn a linear projection that captures semantic similarities between instances. We find that minor modifications, such as adjustments to the learning objectives and the integration of deep learning architectures, significantly enhance NCA's performance. We also introduce a neighbor sampling strategy that improves both the efficiency and predictive accuracy of our proposed ModernNCA.
  arXiv  Detail & Related papers  (2024-07-03T16:38:57Z)
• A Closer Look at Deep Learning on Tabular Data [52.50778536274327]
  Tabular data is prevalent across various domains in machine learning. Deep Neural Network (DNN)-based methods have shown promising performance comparable to tree-based ones.
  arXiv  Detail & Related papers  (2024-07-01T04:24:07Z)
• MambaTab: A Plug-and-Play Model for Learning Tabular Data [13.110156202816112]
  This work introduces an innovative approach based on a structured state-space model (SSM), MambaTab, for tabular data. MambaTab delivers superior performance while requiring significantly fewer parameters, as empirically validated on diverse benchmark datasets.
  arXiv  Detail & Related papers  (2024-01-16T22:44:12Z)
• Graph Neural Network contextual embedding for Deep Learning on Tabular Data [0.45880283710344055]
  Deep Learning (DL) has constituted a major breakthrough for AI in fields related to human skills like natural language processing. This paper presents a novel DL model using Graph Neural Network (GNN) more specifically Interaction Network (IN) Its results outperform those of a recently published survey with DL benchmark based on five public datasets, also achieving competitive results when compared to boosted-tree solutions.
  arXiv  Detail & Related papers  (2023-03-11T17:13:24Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• Transfer Learning with Deep Tabular Models [66.67017691983182]
  We show that upstream data gives tabular neural networks a decisive advantage over GBDT models. We propose a realistic medical diagnosis benchmark for tabular transfer learning. We propose a pseudo-feature method for cases where the upstream and downstream feature sets differ.
  arXiv  Detail & Related papers  (2022-06-30T14:24:32Z)
• Hopular: Modern Hopfield Networks for Tabular Data [5.470026407471584]
  We suggest "Hopular", a novel Deep Learning architecture for medium- and small-sized datasets. Hopular uses stored data to identify feature-feature, feature-target, and sample-sample dependencies. In experiments on small-sized datasets with less than 1,000 samples, Hopular surpasses Gradient Boosting, Random Forests, SVMs, and in particular several Deep Learning methods.
  arXiv  Detail & Related papers  (2022-06-01T17:57:44Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.