Is Deep Learning finally better than Decision Trees on Tabular Data?

• URL: http://arxiv.org/abs/2402.03970v2
• Date: Fri, 14 Feb 2025 14:37:07 GMT
• Title: Is Deep Learning finally better than Decision Trees on Tabular Data?
• Authors: Guri Zabërgja, Arlind Kadra, Christian M. M. Frey, Josif Grabocka,
• Abstract summary: Tabular data is a ubiquitous data modality due to its versatility and ease of use in many real-world applications. Recent studies on data offer a unique perspective on the limitations of neural networks in this domain. Our study categorizes ten state-of-the-art models based on their underlying learning paradigm.
• Score: 19.657605376506357
• License:
• Abstract: Tabular data is a ubiquitous data modality due to its versatility and ease of use in many real-world applications. The predominant heuristics for handling classification tasks on tabular data rely on classical machine learning techniques, as the superiority of deep learning models has not yet been demonstrated. This raises the question of whether new deep learning paradigms can surpass classical approaches. Recent studies on tabular data offer a unique perspective on the limitations of neural networks in this domain and highlight the superiority of gradient boosted decision trees (GBDTs) in terms of scalability and robustness across various datasets. However, novel foundation models have not been thoroughly assessed regarding quality or fairly compared to existing methods for tabular classification. Our study categorizes ten state-of-the-art neural models based on their underlying learning paradigm, demonstrating specifically that meta-learned foundation models outperform GBDTs in small data regimes. Although dataset-specific neural networks generally outperform LLM-based tabular classifiers, they are surpassed by an AutoML library which exhibits the best performance but at the cost of higher computational demands.

Related papers

• Escaping the Forest: Sparse Interpretable Neural Networks for Tabular Data [0.0]
  We show that our models, Sparse TABular NET or sTAB-Net with attention mechanisms, are more effective than tree-based models. They achieve better performance than post-hoc methods like SHAP.
  arXiv  Detail & Related papers  (2024-10-23T10:50:07Z)
• Modern Neighborhood Components Analysis: A Deep Tabular Baseline Two Decades Later [59.88557193062348]
  We revisit the classic Neighborhood Component Analysis (NCA), designed to learn a linear projection that captures semantic similarities between instances. We find that minor modifications, such as adjustments to the learning objectives and the integration of deep learning architectures, significantly enhance NCA's performance. We also introduce a neighbor sampling strategy that improves both the efficiency and predictive accuracy of our proposed ModernNCA.
  arXiv  Detail & Related papers  (2024-07-03T16:38:57Z)
• A Closer Look at Deep Learning Methods on Tabular Datasets [52.50778536274327]
  Tabular data is prevalent across diverse domains in machine learning. Deep Neural Network (DNN)-based methods have recently demonstrated promising performance. We compare 32 state-of-the-art deep and tree-based methods, evaluating their average performance across multiple criteria.
  arXiv  Detail & Related papers  (2024-07-01T04:24:07Z)
• Homological Convolutional Neural Networks [4.615338063719135]
  We propose a novel deep learning architecture that exploits the data structural organization through topologically constrained network representations. We test our model on 18 benchmark datasets against 5 classic machine learning and 3 deep learning models.
  arXiv  Detail & Related papers  (2023-08-26T08:48:51Z)
• NCART: Neural Classification and Regression Tree for Tabular Data [0.5439020425819]
  NCART is a modified version of Residual Networks that replaces fully-connected layers with multiple differentiable oblivious decision trees. It maintains its interpretability while benefiting from the end-to-end capabilities of neural networks. The simplicity of the NCART architecture makes it well-suited for datasets of varying sizes.
  arXiv  Detail & Related papers  (2023-07-23T01:27:26Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• Why do tree-based models still outperform deep learning on tabular data? [0.0]
  We show that tree-based models remain state-of-the-art on medium-sized data. We conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs)
  arXiv  Detail & Related papers  (2022-07-18T08:36:08Z)
• Transfer Learning with Deep Tabular Models [66.67017691983182]
  We show that upstream data gives tabular neural networks a decisive advantage over GBDT models. We propose a realistic medical diagnosis benchmark for tabular transfer learning. We propose a pseudo-feature method for cases where the upstream and downstream feature sets differ.
  arXiv  Detail & Related papers  (2022-06-30T14:24:32Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Model-Based Deep Learning [155.063817656602]
  Signal processing, communications, and control have traditionally relied on classical statistical modeling techniques. Deep neural networks (DNNs) use generic architectures which learn to operate from data, and demonstrate excellent performance. We are interested in hybrid techniques that combine principled mathematical models with data-driven systems to benefit from the advantages of both approaches.
  arXiv  Detail & Related papers  (2020-12-15T16:29:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.