TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling

• URL: http://arxiv.org/abs/2410.24210v2
• Date: Mon, 11 Nov 2024 18:46:06 GMT
• Title: TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling
• Authors: Yury Gorishniy, Akim Kotelnikov, Artem Babenko,
• Abstract summary: We start by developing TabM -- a simple model based on and our variations of BatchEnembles (an existing technique) In particular, we find that TabM demonstrates the best performance among tabular DL models.
• Score: 28.37672139176765
• License:
• Abstract: Deep learning architectures for supervised learning on tabular data range from simple multilayer perceptrons (MLP) to sophisticated Transformers and retrieval-augmented methods. This study highlights a major, yet so far overlooked opportunity for substantially improving tabular MLPs: namely, parameter-efficient ensembling -- a paradigm for implementing an ensemble of models as one model producing multiple predictions. We start by developing TabM -- a simple model based on MLP and our variations of BatchEnsemble (an existing technique). Then, we perform a large-scale evaluation of tabular DL architectures on public benchmarks in terms of both task performance and efficiency, which renders the landscape of tabular DL in a new light. Generally, we show that MLPs, including TabM, form a line of stronger and more practical models compared to attention- and retrieval-based architectures. In particular, we find that TabM demonstrates the best performance among tabular DL models. Lastly, we conduct an empirical analysis on the ensemble-like nature of TabM. For example, we observe that the multiple predictions of TabM are weak individually, but powerful collectively. Overall, our work brings an impactful technique to tabular DL, analyses its behaviour, and advances the performance-efficiency trade-off with TabM -- a simple and powerful baseline for researchers and practitioners.

Related papers

• Towards Better Understanding Table Instruction Tuning: Decoupling the Effects from Data versus Models [62.47618742274461]
  We fine-tune base models from the Mistral, OLMo, and Phi families on existing public training datasets. Our replication achieves performance on par with or surpassing existing table LLMs. We decouple the contributions of training data and the base model, providing insight into their individual impacts.
  arXiv  Detail & Related papers  (2025-01-24T18:50:26Z)
• Rethinking Table Instruction Tuning [29.139828718538418]
  We evaluate abilities in existing table LLMs and reveal significant declines in both out-of-domain table understanding and general capabilities. We introduce TAMA, a TAble LLM instruction-tuned from LLaMA 3.1 8B Instruct, which achieves performance on par with, or surpassing GPT-3.5 and GPT-4 on table tasks.
  arXiv  Detail & Related papers  (2025-01-24T18:06:07Z)
• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale [66.73529246309033]
  multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. Existing instruction-tuning datasets only provide phrase-level answers without any intermediate rationales. We introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales.
  arXiv  Detail & Related papers  (2024-12-06T18:14:24Z)
• TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios [51.66718740300016]
  TableLLM is a robust large language model (LLM) with 8 billion parameters. TableLLM is purpose-built for proficiently handling data manipulation tasks. We have released the model checkpoint, source code, benchmarks, and a web application for user interaction.
  arXiv  Detail & Related papers  (2024-03-28T11:21:12Z)
• Deep Learning with Tabular Data: A Self-supervised Approach [0.0]
  We have used a self-supervised learning approach in this study. The aim is to find the most effective TabTransformer model representation of categorical and numerical features. The research has presented with a novel approach by creating various variants of TabTransformer model.
  arXiv  Detail & Related papers  (2024-01-26T23:12:41Z)
• MambaTab: A Plug-and-Play Model for Learning Tabular Data [13.110156202816112]
  This work introduces an innovative approach based on a structured state-space model (SSM), MambaTab, for tabular data. MambaTab delivers superior performance while requiring significantly fewer parameters, as empirically validated on diverse benchmark datasets.
  arXiv  Detail & Related papers  (2024-01-16T22:44:12Z)
• TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning [55.33939289989238]
  We propose TAP4LLM as a versatile pre-processor suite for leveraging large language models (LLMs) in table-based tasks effectively. It covers several distinct components: (1) table sampling to decompose large tables into manageable sub-tables based on query semantics, (2) table augmentation to enhance tables with additional knowledge from external sources or models, and (3) table packing & serialization to convert tables into various formats suitable for LLMs' understanding.
  arXiv  Detail & Related papers  (2023-12-14T15:37:04Z)
• UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science [16.384705926693073]
  This study seeks to extend the power of pretraining methodologies to facilitate the prediction over tables in data science. We introduce UniTabE, a method designed to process tables in a uniform manner, devoid of constraints imposed by specific table structures. In order to implement the pretraining phase, we curated an expansive dataset comprising approximately 13B samples, meticulously gathered from the Kaggle platform.
  arXiv  Detail & Related papers  (2023-07-18T13:28:31Z)
• Language models are weak learners [71.33837923104808]
  We show that prompt-based large language models can operate effectively as weak learners. We incorporate these models into a boosting approach, which can leverage the knowledge within the model to outperform traditional tree-based boosting. Results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
  arXiv  Detail & Related papers  (2023-06-25T02:39:19Z)
• STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables [64.0903766169603]
  We propose a framework for few-shot semi-supervised learning, coined Self-generated Tasks from UNlabeled Tables (STUNT) Our key idea is to self-generate diverse few-shot tasks by treating randomly chosen columns as a target label. We then employ a meta-learning scheme to learn generalizable knowledge with the constructed tasks.
  arXiv  Detail & Related papers  (2023-03-02T02:37:54Z)
• The GatedTabTransformer. An enhanced deep learning architecture for tabular modeling [0.0]
  We propose multiple modifications to the original TabTransformer performing better on binary classification tasks. Inspired by gated, linear projections are implemented in the block and multiple activation functions are tested.
  arXiv  Detail & Related papers  (2022-01-01T14:52:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.