Generative Table Pre-training Empowers Models for Tabular Prediction
- URL: http://arxiv.org/abs/2305.09696v1
- Date: Tue, 16 May 2023 06:37:38 GMT
- Title: Generative Table Pre-training Empowers Models for Tabular Prediction
- Authors: Tianping Zhang, Shaowen Wang, Shuicheng Yan, Jian Li, Qian Liu
- Abstract summary: We propose TapTap, the first attempt that leverages table pre-training to empower models for tabular prediction.
TapTap can generate high-quality synthetic tables to support various applications, including privacy protection, low resource regime, missing value imputation, and imbalanced classification.
It can be easily combined with various backbone models, including LightGBM, Multilayer Perceptron (MLP) and Transformer.
- Score: 71.76829961276032
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, the topic of table pre-training has attracted considerable research
interest. However, how to employ table pre-training to boost the performance of
tabular prediction remains an open challenge. In this paper, we propose TapTap,
the first attempt that leverages table pre-training to empower models for
tabular prediction. After pre-training on a large corpus of real-world tabular
data, TapTap can generate high-quality synthetic tables to support various
applications on tabular data, including privacy protection, low resource
regime, missing value imputation, and imbalanced classification. Extensive
experiments on 12 datasets demonstrate that TapTap outperforms a total of 16
baselines in different scenarios. Meanwhile, it can be easily combined with
various backbone models, including LightGBM, Multilayer Perceptron (MLP) and
Transformer. Moreover, with the aid of table pre-training, models trained using
synthetic data generated by TapTap can even compete with models using the
original dataset on half of the experimental datasets, marking a milestone in
the development of synthetic tabular data generation. The codes are available
at https://github.com/ZhangTP1996/TapTap.
Related papers
- TabDPT: Scaling Tabular Foundation Models [20.00390825519329]
We show how to harness the power of real data to improve performance and generalization.
Our model achieves state-of-the-art performance on the CC18 (classification) and CTR23 (regression) benchmarks.
TabDPT also demonstrates strong scaling as both model size and amount of available data increase.
arXiv Detail & Related papers (2024-10-23T18:00:00Z) - LaTable: Towards Large Tabular Models [63.995130144110156]
Tabular generative foundation models are hard to build due to the heterogeneous feature spaces of different datasets.
LaTable is a novel diffusion model that addresses these challenges and can be trained across different datasets.
We find that LaTable outperforms baselines on in-distribution generation, and that finetuning LaTable can generate out-of-distribution datasets better with fewer samples.
arXiv Detail & Related papers (2024-06-25T16:03:50Z) - Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
The transferability of deep neural networks (DNNs) has made significant progress in image and language processing.
We present TP-BERTa, a specifically pre-trained LM for tabular data prediction.
A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
arXiv Detail & Related papers (2024-03-04T08:38:56Z) - Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective [71.45945607871715]
We propose Tabular data Pre-Training via Meta-representation (TabPTM)
The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels.
Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
arXiv Detail & Related papers (2023-10-31T18:03:54Z) - TabuLa: Harnessing Language Models for Tabular Data Synthesis [4.539846270369207]
Tabula is a tabular data synthesizer that leverages the structure of large language models (LLMs)
Unlike state-of-the-art (SOTA) LLMs, Tabula discards the pre-trained weights originally designed for natural language tasks.
experiments show that Tabula achieves superior synthetic data utility compared to current SOTA methods.
arXiv Detail & Related papers (2023-10-19T13:50:56Z) - UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model
in Data Science [16.384705926693073]
This study seeks to extend the power of pretraining methodologies to facilitate the prediction over tables in data science.
We introduce UniTabE, a method designed to process tables in a uniform manner, devoid of constraints imposed by specific table structures.
In order to implement the pretraining phase, we curated an expansive dataset comprising approximately 13B samples, meticulously gathered from the Kaggle platform.
arXiv Detail & Related papers (2023-07-18T13:28:31Z) - OmniTab: Pretraining with Natural and Synthetic Data for Few-shot
Table-based Question Answering [106.73213656603453]
We develop a simple table-based QA model with minimal annotation effort.
We propose an omnivorous pretraining approach that consumes both natural and synthetic data.
arXiv Detail & Related papers (2022-07-08T01:23:45Z) - GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing [117.98107557103877]
We present GraPPa, an effective pre-training approach for table semantic parsing.
We construct synthetic question-pairs over high-free tables via a synchronous context-free grammar.
To maintain the model's ability to represent real-world data, we also include masked language modeling.
arXiv Detail & Related papers (2020-09-29T08:17:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.