Mambular: A Sequential Model for Tabular Deep Learning

• URL: http://arxiv.org/abs/2408.06291v2
• Date: Tue, 25 Mar 2025 17:27:53 GMT
• Title: Mambular: A Sequential Model for Tabular Deep Learning
• Authors: Anton Frederik Thielmann, Manish Kumar, Christoph Weisser, Arik Reuter, Benjamin Säfken, Soheila Samiee,
• Abstract summary: This paper investigates the use of autoregressive state-space models for tabular data.<n>We compare their performance against established benchmark models.<n>Our findings indicate that interpreting features as a sequence and processing them can lead to significant performance improvement.
• Score: 0.7184556517162347
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The analysis of tabular data has traditionally been dominated by gradient-boosted decision trees (GBDTs), known for their proficiency with mixed categorical and numerical features. However, recent deep learning innovations are challenging this dominance. This paper investigates the use of autoregressive state-space models for tabular data and compares their performance against established benchmark models. Additionally, we explore various adaptations of these models, including different pooling strategies, feature interaction mechanisms, and bi-directional processing techniques to understand their effectiveness for tabular data. Our findings indicate that interpreting features as a sequence and processing them and their interactions through structured state-space layers can lead to significant performance improvement. This research underscores the versatility of autoregressive models in tabular data analysis, positioning them as a promising alternative that could substantially enhance deep learning capabilities in this traditionally challenging area. The source code is available at https://github.com/basf/mamba-tabular.

Related papers

• TabKAN: Advancing Tabular Data Analysis using Kolmograv-Arnold Network [11.664880068737084]
  This paper introduces TabKAN, a novel framework that advances tabular data modeling using Kolmogorov-Arnold Networks (KANs) KANs leverage learnable activation functions on edges, enhancing both interpretability and training efficiency. Through extensive benchmarking on diverse public datasets, TabKAN demonstrates superior performance in supervised learning while significantly outperforming classical and Transformer-based models in transfer learning scenarios.
  arXiv  Detail & Related papers  (2025-04-09T03:46:10Z)
• In-context learning of evolving data streams with tabular foundational models [42.13420474990124]
  This work bridges advancements from both areas, highlighting how transformers' implicit meta-learning abilities, pre-training on drifting natural data, and reliance on context optimization directly address the core challenges of adaptive learning in dynamic environments. Exploring real-time model adaptation, this research demonstrates that TabPFN, coupled with a simple sliding memory strategy, consistently outperforms ensembles of Hoeffding trees across all non-stationary benchmarks.
  arXiv  Detail & Related papers  (2025-02-24T04:52:35Z)
• Escaping the Forest: Sparse Interpretable Neural Networks for Tabular Data [0.0]
  We show that our models, Sparse TABular NET or sTAB-Net with attention mechanisms, are more effective than tree-based models. They achieve better performance than post-hoc methods like SHAP.
  arXiv  Detail & Related papers  (2024-10-23T10:50:07Z)
• Modern Neighborhood Components Analysis: A Deep Tabular Baseline Two Decades Later [59.88557193062348]
  We revisit the classic Neighborhood Component Analysis (NCA), designed to learn a linear projection that captures semantic similarities between instances. We find that minor modifications, such as adjustments to the learning objectives and the integration of deep learning architectures, significantly enhance NCA's performance. We also introduce a neighbor sampling strategy that improves both the efficiency and predictive accuracy of our proposed ModernNCA.
  arXiv  Detail & Related papers  (2024-07-03T16:38:57Z)
• A Closer Look at Deep Learning on Tabular Data [52.50778536274327]
  Tabular data is prevalent across various domains in machine learning. Deep Neural Network (DNN)-based methods have shown promising performance comparable to tree-based ones.
  arXiv  Detail & Related papers  (2024-07-01T04:24:07Z)
• Is Deep Learning finally better than Decision Trees on Tabular Data? [19.657605376506357]
  Tabular data is a ubiquitous data modality due to its versatility and ease of use in many real-world applications. Recent studies on data offer a unique perspective on the limitations of neural networks in this domain. Our study categorizes ten state-of-the-art models based on their underlying learning paradigm.
  arXiv  Detail & Related papers  (2024-02-06T12:59:02Z)
• MambaTab: A Plug-and-Play Model for Learning Tabular Data [13.110156202816112]
  This work introduces an innovative approach based on a structured state-space model (SSM), MambaTab, for tabular data. MambaTab delivers superior performance while requiring significantly fewer parameters, as empirically validated on diverse benchmark datasets.
  arXiv  Detail & Related papers  (2024-01-16T22:44:12Z)
• Graph Neural Network contextual embedding for Deep Learning on Tabular Data [0.45880283710344055]
  Deep Learning (DL) has constituted a major breakthrough for AI in fields related to human skills like natural language processing. This paper presents a novel DL model using Graph Neural Network (GNN) more specifically Interaction Network (IN) Its results outperform those of a recently published survey with DL benchmark based on five public datasets, also achieving competitive results when compared to boosted-tree solutions.
  arXiv  Detail & Related papers  (2023-03-11T17:13:24Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• Transfer Learning with Deep Tabular Models [66.67017691983182]
  We show that upstream data gives tabular neural networks a decisive advantage over GBDT models. We propose a realistic medical diagnosis benchmark for tabular transfer learning. We propose a pseudo-feature method for cases where the upstream and downstream feature sets differ.
  arXiv  Detail & Related papers  (2022-06-30T14:24:32Z)
• Hopular: Modern Hopfield Networks for Tabular Data [5.470026407471584]
  We suggest "Hopular", a novel Deep Learning architecture for medium- and small-sized datasets. Hopular uses stored data to identify feature-feature, feature-target, and sample-sample dependencies. In experiments on small-sized datasets with less than 1,000 samples, Hopular surpasses Gradient Boosting, Random Forests, SVMs, and in particular several Deep Learning methods.
  arXiv  Detail & Related papers  (2022-06-01T17:57:44Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.