Tree-Regularized Tabular Embeddings
- URL: http://arxiv.org/abs/2403.00963v1
- Date: Fri, 1 Mar 2024 20:26:33 GMT
- Title: Tree-Regularized Tabular Embeddings
- Authors: Xuan Li, Yun Wang, Bo Li
- Abstract summary: Tabular neural network (NN) has attracted remarkable attentions and its recent advances have gradually narrowed the performance gap with respect to tree-based models on many public datasets.
We emphasize the importance of homogeneous embeddings and alternately concentrate on regularizing inputs through supervised pretraining.
Specifically, we utilize the structure of pretrained tree ensembles to transform raw variables into a single vector (T2V), or an array of tokens (T2T)
- Score: 22.095328171882223
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Tabular neural network (NN) has attracted remarkable attentions and its
recent advances have gradually narrowed the performance gap with respect to
tree-based models on many public datasets. While the mainstreams focus on
calibrating NN to fit tabular data, we emphasize the importance of homogeneous
embeddings and alternately concentrate on regularizing tabular inputs through
supervised pretraining. Specifically, we extend a recent work (DeepTLF) and
utilize the structure of pretrained tree ensembles to transform raw variables
into a single vector (T2V), or an array of tokens (T2T). Without loss of space
efficiency, these binarized embeddings can be consumed by canonical tabular NN
with fully-connected or attention-based building blocks. Through quantitative
experiments on 88 OpenML datasets with binary classification task, we validated
that the proposed tree-regularized representation not only tapers the
difference with respect to tree-based models, but also achieves on-par and
better performance when compared with advanced NN models. Most importantly, it
possesses better robustness and can be easily scaled and generalized as
standalone encoder for tabular modality. Codes:
https://github.com/milanlx/tree-regularized-embedding.
Related papers
- Rapid and Precise Topological Comparison with Merge Tree Neural Networks [7.443474354626664]
We introduce the merge tree neural networks (MTNN), a learned neural network model designed for merge tree comparison.
In particular, we speed up the prior state-of-the-art by more than 100x on the benchmark datasets while maintaining an error rate below 0.1%.
arXiv Detail & Related papers (2024-04-08T21:26:04Z) - Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
The transferability of deep neural networks (DNNs) has made significant progress in image and language processing.
We present TP-BERTa, a specifically pre-trained LM for tabular data prediction.
A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
arXiv Detail & Related papers (2024-03-04T08:38:56Z) - Boosting gets full Attention for Relational Learning [27.82663283409287]
We introduce an attention mechanism for structured data that blends well with tree-based models in the training context of (gradient) boosting.
Experiments on simulated and real-world domains display the competitiveness of our method against a state of the art containing both tree-based and neural nets-based models.
arXiv Detail & Related papers (2024-02-22T19:16:01Z) - TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks [90.00817095558094]
We develop context optimization techniques for prior-data fitted networks (PFNs)
PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass.
We propose TuneTables, a novel prompt-tuning strategy that compresses large datasets into a smaller learned context.
arXiv Detail & Related papers (2024-02-17T00:02:23Z) - Efficient Link Prediction via GNN Layers Induced by Negative Sampling [92.05291395292537]
Graph neural networks (GNNs) for link prediction can loosely be divided into two broad categories.
First, emphnode-wise architectures pre-compute individual embeddings for each node that are later combined by a simple decoder to make predictions.
Second, emphedge-wise methods rely on the formation of edge-specific subgraph embeddings to enrich the representation of pair-wise relationships.
arXiv Detail & Related papers (2023-10-14T07:02:54Z) - Compacting Binary Neural Networks by Sparse Kernel Selection [58.84313343190488]
This paper is motivated by a previously revealed phenomenon that the binary kernels in successful BNNs are nearly power-law distributed.
We develop the Permutation Straight-Through Estimator (PSTE) that is able to not only optimize the selection process end-to-end but also maintain the non-repetitive occupancy of selected codewords.
Experiments verify that our method reduces both the model size and bit-wise computational costs, and achieves accuracy improvements compared with state-of-the-art BNNs under comparable budgets.
arXiv Detail & Related papers (2023-03-25T13:53:02Z) - Sparse tree-based initialization for neural networks [0.0]
We show that dedicated neural network (NN) architectures can handle specific data types such as CNN for images or RNN for text.
In this work, we propose a new technique for (potentially deep) multilayer perceptrons (MLP)
We show that our new initializer operates an implicit regularization during the NN training, and emphasizes that the first layers act as a sparse feature extractor.
arXiv Detail & Related papers (2022-09-30T07:44:03Z) - Recurrent Bilinear Optimization for Binary Neural Networks [58.972212365275595]
BNNs neglect the intrinsic bilinear relationship of real-valued weights and scale factors.
Our work is the first attempt to optimize BNNs from the bilinear perspective.
We obtain robust RBONNs, which show impressive performance over state-of-the-art BNNs on various models and datasets.
arXiv Detail & Related papers (2022-09-04T06:45:33Z) - Why do tree-based models still outperform deep learning on tabular data? [0.0]
We show that tree-based models remain state-of-the-art on medium-sized data.
We conduct an empirical investigation into the differing inductive biases of tree-based models and Neural Networks (NNs)
arXiv Detail & Related papers (2022-07-18T08:36:08Z) - Temporal Calibrated Regularization for Robust Noisy Label Learning [60.90967240168525]
Deep neural networks (DNNs) exhibit great success on many tasks with the help of large-scale well annotated datasets.
However, labeling large-scale data can be very costly and error-prone so that it is difficult to guarantee the annotation quality.
We propose a Temporal Calibrated Regularization (TCR) in which we utilize the original labels and the predictions in the previous epoch together.
arXiv Detail & Related papers (2020-07-01T04:48:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.