TabSTAR: A Foundation Tabular Model With Semantically Target-Aware Representations

• URL: http://arxiv.org/abs/2505.18125v1
• Date: Fri, 23 May 2025 17:34:28 GMT
• Title: TabSTAR: A Foundation Tabular Model With Semantically Target-Aware Representations
• Authors: Alan Arazi, Eilam Shapira, Roi Reichart,
• Abstract summary: Tabular Foundation Models can leverage real-world knowledge and generalize across diverse datasets.<n>We introduce TabSTAR: a Foundation Tabular Model with Semantically Target-Aware Representations.
• Score: 14.12892960275563
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While deep learning has achieved remarkable success across many domains, it has historically underperformed on tabular learning tasks, which remain dominated by gradient boosting decision trees (GBDTs). However, recent advancements are paving the way for Tabular Foundation Models, which can leverage real-world knowledge and generalize across diverse datasets, particularly when the data contains free-text. Although incorporating language model capabilities into tabular tasks has been explored, most existing methods utilize static, target-agnostic textual representations, limiting their effectiveness. We introduce TabSTAR: a Foundation Tabular Model with Semantically Target-Aware Representations. TabSTAR is designed to enable transfer learning on tabular data with textual features, with an architecture free of dataset-specific parameters. It unfreezes a pretrained text encoder and takes as input target tokens, which provide the model with the context needed to learn task-specific embeddings. TabSTAR achieves state-of-the-art performance for both medium- and large-sized datasets across known benchmarks of classification tasks with text features, and its pretraining phase exhibits scaling laws in the number of datasets, offering a pathway for further performance improvements.

Related papers

• Towards Benchmarking Foundation Models for Tabular Data With Text [36.3195231571412]
  We propose a series of simple yet effective ablation-style strategies for incorporating text into conventional pipelines.<n>We benchmark how state-of-the-art tabular foundation models can handle textual data by manually curating a collection of real-world datasets with meaningful textual features.
  arXiv  Detail & Related papers  (2025-07-10T15:01:31Z)
• ConTextTab: A Semantics-Aware Tabular In-Context Learner [0.0]
  We introduce ConTextTab, integrating semantic understanding and alignment into a table-native ICL framework.<n>Our model is competitive with SOTA across a broad set of benchmarks while setting a new standard on the semantically rich CARTE benchmark.
  arXiv  Detail & Related papers  (2025-06-12T13:57:29Z)
• Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard [27.224577475861214]
  Tabular data, a fundamental data format in machine learning, is predominantly utilized in competitions and real-world applications.<n>We propose an in-context ensemble framework for tabular prediction that leverages large language models.<n>Our method constructs a context around each test instance using its nearest neighbors and the predictions from a pool of external models.
  arXiv  Detail & Related papers  (2025-05-19T17:52:58Z)
• Representation Learning for Tabular Data: A Comprehensive Survey [23.606506938919605]
  Tabular data, structured as rows and columns, is among the most prevalent data types in machine learning classification and regression applications.<n>Deep Neural Networks (DNNs) have recently demonstrated promising results through their capability of representation learning.<n>We organize existing methods into three main categories according to their generalization capabilities.
  arXiv  Detail & Related papers  (2025-04-17T17:58:23Z)
• A Closer Look at Deep Learning Methods on Tabular Datasets [52.50778536274327]
  Tabular data is prevalent across diverse domains in machine learning.<n>Deep Neural Network (DNN)-based methods have recently demonstrated promising performance.<n>We compare 32 state-of-the-art deep and tree-based methods, evaluating their average performance across multiple criteria.
  arXiv  Detail & Related papers  (2024-07-01T04:24:07Z)
• Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective [71.45945607871715]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM)<n>The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels.<n>Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• Unlocking the Transferability of Tokens in Deep Models for Tabular Data [67.11727608815636]
  Fine-tuning a pre-trained deep neural network has become a successful paradigm in various machine learning tasks. In this paper, we propose TabToken, a method aims at enhancing the quality of feature tokens. We introduce a contrastive objective that regularizes the tokens, capturing the semantics within and across features.
  arXiv  Detail & Related papers  (2023-10-23T17:53:09Z)
• From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models [18.219485459836285]
  Generative Tabular Learning (GTL) is a novel framework that integrates the advanced functionalities of large language models (LLMs) Our empirical study spans 384 public datasets, rigorously analyzing GTL's scaling behaviors. GTL-LLaMA-2 model demonstrates superior zero-shot and in-context learning capabilities across numerous classification and regression tasks.
  arXiv  Detail & Related papers  (2023-10-11T09:37:38Z)
• infoVerse: A Universal Framework for Dataset Characterization with Multidimensional Meta-information [68.76707843019886]
  infoVerse is a universal framework for dataset characterization. infoVerse captures multidimensional characteristics of datasets by incorporating various model-driven meta-information. In three real-world applications (data pruning, active learning, and data annotation), the samples chosen on infoVerse space consistently outperform strong baselines.
  arXiv  Detail & Related papers  (2023-05-30T18:12:48Z)
• STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables [64.0903766169603]
  We propose a framework for few-shot semi-supervised learning, coined Self-generated Tasks from UNlabeled Tables (STUNT) Our key idea is to self-generate diverse few-shot tasks by treating randomly chosen columns as a target label. We then employ a meta-learning scheme to learn generalizable knowledge with the constructed tasks.
  arXiv  Detail & Related papers  (2023-03-02T02:37:54Z)
• TabLLM: Few-shot Classification of Tabular Data with Large Language Models [66.03023402174138]
  We study the application of large language models to zero-shot and few-shot classification. We evaluate several serialization methods including templates, table-to-text models, and large language models. This approach is also competitive with strong traditional baselines like gradient-boosted trees.
  arXiv  Detail & Related papers  (2022-10-19T17:08:13Z)
• SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning [5.5616364225463055]
  We introduce a new framework, Subsetting features of Tabular data (SubTab) In this paper, we introduce a new framework, Subsetting features of Tabular data (SubTab) We argue that reconstructing the data from the subset of its features rather than its corrupted version in an autoencoder setting can better capture its underlying representation.
  arXiv  Detail & Related papers  (2021-10-08T20:11:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.