Tab-PET: Graph-Based Positional Encodings for Tabular Transformers

• URL: http://arxiv.org/abs/2511.13338v1
• Date: Mon, 17 Nov 2025 13:08:34 GMT
• Title: Tab-PET: Graph-Based Positional Encodings for Tabular Transformers
• Authors: Yunze Leng, Rohan Ghosh, Mehul Motani,
• Abstract summary: We propose a graph-based framework for estimating and inculcating positional encodings into embeddings.<n>Inspired by approaches that derive PEs from graph topology, we explore two paradigms for graph estimation: association-based and causality-based.<n>We empirically demonstrate that graph-derived PEs significantly improve performance across 50 classification and regression datasets for 3T. Notably, association-based graphs consistently yield more stable and pronounced gains compared to causality-driven ones.
• Score: 10.312868216225118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Supervised learning with tabular data presents unique challenges, including low data sizes, the absence of structural cues, and heterogeneous features spanning both categorical and continuous domains. Unlike vision and language tasks, where models can exploit inductive biases in the data, tabular data lacks inherent positional structure, hindering the effectiveness of self-attention mechanisms. While recent transformer-based models like TabTransformer, SAINT, and FT-Transformer (which we refer to as 3T) have shown promise on tabular data, they typically operate without leveraging structural cues such as positional encodings (PEs), as no prior structural information is usually available. In this work, we find both theoretically and empirically that structural cues, specifically PEs can be a useful tool to improve generalization performance for tabular transformers. We find that PEs impart the ability to reduce the effective rank (a form of intrinsic dimensionality) of the features, effectively simplifying the task by reducing the dimensionality of the problem, yielding improved generalization. To that end, we propose Tab-PET (PEs for Tabular Transformers), a graph-based framework for estimating and inculcating PEs into embeddings. Inspired by approaches that derive PEs from graph topology, we explore two paradigms for graph estimation: association-based and causality-based. We empirically demonstrate that graph-derived PEs significantly improve performance across 50 classification and regression datasets for 3T. Notably, association-based graphs consistently yield more stable and pronounced gains compared to causality-driven ones. Our work highlights an unexpected role of PEs in tabular transformers, revealing how they can be harnessed to improve generalization.

Related papers

• Plain Transformers Can be Powerful Graph Learners [64.50059165186701]
  Researchers have attempted to migrate Transformers to graph learning, but most advanced Graph Transformers have strayed far from plain Transformers.<n>This work demonstrates that the plain Transformer architecture can be a powerful graph learner.
  arXiv  Detail & Related papers  (2025-04-17T02:06:50Z)
• A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities [51.08999772842298]
  Tabular Prior-data Fitted Network v2 (TabPFN v2) achieves unprecedented in-context learning performance across diverse downstream datasets.<n>We show that TabPFN v2 can infer attribute relationships even when provided with randomized attribute token inputs.<n>We demonstrate that TabPFN v2's limitations can be addressed through a test-time divide-and-context strategy.
  arXiv  Detail & Related papers  (2025-02-24T17:38:42Z)
• TabTreeFormer: Tabular Data Generation Using Hybrid Tree-Transformer [14.330758748478281]
  TabTreeFormer is a hybrid transformer architecture that integrates inductive biases of tree-based models.<n>We show that TabTreeFormer consistently outperforms baselines in utility, fidelity, and privacy metrics with competitive efficiency.
  arXiv  Detail & Related papers  (2025-01-02T11:57:08Z)
• Structural Entropy Guided Probabilistic Coding [52.01765333755793]
  We propose a novel structural entropy-guided probabilistic coding model, named SEPC.<n>We incorporate the relationship between latent variables into the optimization by proposing a structural entropy regularization loss.<n> Experimental results across 12 natural language understanding tasks, including both classification and regression tasks, demonstrate the superior performance of SEPC.
  arXiv  Detail & Related papers  (2024-12-12T00:37:53Z)
• A Survey on Deep Tabular Learning [0.0]
  Tabular data presents unique challenges for deep learning due to its heterogeneous nature and lack of spatial structure. This survey reviews the evolution of deep learning models for Tabular data, from early fully connected networks (FCNs) to advanced architectures like TabNet, SAINT, TabTranSELU, and MambaNet.
  arXiv  Detail & Related papers  (2024-10-15T20:08:08Z)
• Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
  The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. We present TP-BERTa, a specifically pre-trained LM for tabular data prediction. A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
  arXiv  Detail & Related papers  (2024-03-04T08:38:56Z)
• Unifying Structured Data as Graph for Data-to-Text Pre-Training [69.96195162337793]
  Data-to-text (D2T) generation aims to transform structured data into natural language text. Data-to-text pre-training has proved to be powerful in enhancing D2T generation. We propose a structure-enhanced pre-training method for D2T generation by designing a structure-enhanced Transformer.
  arXiv  Detail & Related papers  (2024-01-02T12:23:49Z)
• Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective [71.45945607871715]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM)<n>The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels.<n>Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networks [0.7118812771905295]
  We show that it is sub-optimal to directly transfer existing PEFTs to graph-based tasks due to the issue of feature distribution shift. We propose a novel structure-aware PEFT approach, named G-Adapter, to guide the updating process. Extensive experiments demonstrate that G-Adapter obtains the state-of-the-art performance compared to the counterparts on nine graph benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-17T16:10:36Z)
• Named Entity Recognition in Industrial Tables using Tabular Language Models [24.287536314062965]
  We study how these models can be applied to an industrial Named Entity Recognition (NER) problem. The highly technical nature of spreadsheets as well as the lack of labeled data present major challenges for fine-tuning transformer-based models.
  arXiv  Detail & Related papers  (2022-09-29T14:25:44Z)
• Learning Enhanced Representations for Tabular Data via Neighborhood Propagation [24.485479610138498]
  We construct a hypergraph to model the cross-row and cross-column patterns of data instances. We then perform message propagation to enhance the target data instance representation. Experiments on two important data prediction tasks validate the superiority of the proposed PET model.
  arXiv  Detail & Related papers  (2022-06-14T04:24:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.