Tokenize features, enhancing tables: the FT-TABPFN model for tabular classification

• URL: http://arxiv.org/abs/2406.06891v1
• Date: Tue, 11 Jun 2024 02:13:46 GMT
• Title: Tokenize features, enhancing tables: the FT-TABPFN model for tabular classification
• Authors: Quangao Liu, Wei Yang, Chen Liang, Longlong Pang, Zhuozhang Zou,
• Abstract summary: FT-TabPFN is an enhanced version of TabPFN that includes a novel Feature Tokenization layer to better handle classification features. Our full source code is available for community use and development.
• Score: 13.481699494376809
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional methods for tabular classification usually rely on supervised learning from scratch, which requires extensive training data to determine model parameters. However, a novel approach called Prior-Data Fitted Networks (TabPFN) has changed this paradigm. TabPFN uses a 12-layer transformer trained on large synthetic datasets to learn universal tabular representations. This method enables fast and accurate predictions on new tasks with a single forward pass and no need for additional training. Although TabPFN has been successful on small datasets, it generally shows weaker performance when dealing with categorical features. To overcome this limitation, we propose FT-TabPFN, which is an enhanced version of TabPFN that includes a novel Feature Tokenization layer to better handle classification features. By fine-tuning it for downstream tasks, FT-TabPFN not only expands the functionality of the original model but also significantly improves its applicability and accuracy in tabular classification. Our full source code is available for community use and development.

Related papers

• TabPFGen -- Tabular Data Generation with TabPFN [4.743548909570325]
  We turn TabPFN, a highly performant transformer, into an energy-based generative model, which we dub TabPFGen. We demonstrate strong results on standard generative modelling tasks, including data augmentation, class-balancing, and imputation.
  arXiv  Detail & Related papers  (2024-06-07T18:59:37Z)
• Interpretable Machine Learning for TabPFN [5.012821694203072]
  The TabPFN model is able to achieve state-of-the-art performance on a variety of classification tasks. By taking advantage of the unique properties of the model, our adaptations allow for more efficient computations.
  arXiv  Detail & Related papers  (2024-03-16T13:35:15Z)
• Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
  The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. We present TP-BERTa, a specifically pre-trained LM for tabular data prediction. A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
  arXiv  Detail & Related papers  (2024-03-04T08:38:56Z)
• TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks [90.00817095558094]
  Prior-data fitted networks (PFNs) make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. We introduce TuneTables, a parameter-efficient fine-tuning strategy for PFNs that compresses large datasets into a smaller learned context. We show that TuneTables can be used as an interpretability tool and can even be used to mitigate biases by optimizing a fairness objective.
  arXiv  Detail & Related papers  (2024-02-17T00:02:23Z)
• In-Context Data Distillation with TabPFN [11.553950697974825]
  In-context data distillation (ICD) is a novel methodology that effectively eliminates these constraints by optimizing TabPFN's context. ICD efficiently enables TabPFN to handle significantly larger datasets with a fixed memory budget, improving TabPFN's quadratic memory complexity but at the cost of a linear number of tuning steps.
  arXiv  Detail & Related papers  (2024-02-10T15:23:45Z)
• Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks [31.82225213006849]
  Tabular classification has traditionally relied on supervised algorithms, which estimate the parameters of a prediction model using its training data. Recently, Prior-Data Fitted Networks (PFNs) such as TabPFN have successfully learned to classify tabular data in-context. While such models show great promise, their applicability to real-world data remains limited due to the computational scale needed.
  arXiv  Detail & Related papers  (2023-11-17T16:04:27Z)
• Training-Free Generalization on Heterogeneous Tabular Data via Meta-Representation [67.30538142519067]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM) A deep neural network is then trained to associate these meta-representations with dataset-specific classification confidences. Experiments validate that TabPTM achieves promising performance in new datasets, even under few-shot scenarios.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• Unlocking the Transferability of Tokens in Deep Models for Tabular Data [67.11727608815636]
  Fine-tuning a pre-trained deep neural network has become a successful paradigm in various machine learning tasks. In this paper, we propose TabToken, a method aims at enhancing the quality of feature tokens. We introduce a contrastive objective that regularizes the tokens, capturing the semantics within and across features.
  arXiv  Detail & Related papers  (2023-10-23T17:53:09Z)
• Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need [84.3507610522086]
  Class-incremental learning (CIL) aims to adapt to emerging new classes without forgetting old ones. Recent pre-training has achieved substantial progress, making vast pre-trained models (PTMs) accessible for CIL. We argue that the core factors in CIL are adaptivity for model updating and generalizability for knowledge transferring.
  arXiv  Detail & Related papers  (2023-03-13T17:59:02Z)
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second [48.87527918630822]
  We present TabPFN, a trained Transformer that can do supervised classification for small datasets in less than a second. TabPFN performs in-context learning (ICL), it learns to make predictions using sequences of labeled examples. We show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to 230$times$ speedup.
  arXiv  Detail & Related papers  (2022-07-05T07:17:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.