Enhancing Tabular Learners with Context-Aware Semantic Embeddings

• URL: http://arxiv.org/abs/2608.03565v1
• Date: Tue, 04 Aug 2026 12:29:37 GMT
• Title: Enhancing Tabular Learners with Context-Aware Semantic Embeddings
• Abstract summary: We propose a novel framework that bridges the gap between the semantic understanding of Large Language Models (LLMs) and the statistical capabilities of tabular learners.<n>Case utilizes a contextualization strategy: we pre-fill the KV cache of a custom-trained Gemma 3-based Tabular Language Model with a representative sample of rows to establish a persistent anchor of the dataset's semantics.
• Score: 1.6381263995389714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While modern tabular learners excel at capturing statistical patterns, they frequently operate in a semantic vacuum, treating textual features as discrete symbols, ignoring the rich semantics inherent in feature names or cell entries. We propose CASE (Context-Aware Semantic Embeddings), a novel framework that bridges the gap between the semantic understanding of Large Language Models (LLMs) and the statistical capabilities of tabular learners. Unlike existing methods that embed rows in isolation, CASE utilizes a contextualization strategy: we pre-fill the KV cache of a custom-trained Gemma 3-based Tabular Language Model with a representative sample of rows to establish a persistent anchor of the dataset's semantics. This ensures that generated row embeddings are dynamically contextualized, resolving semantic ambiguities and anchoring representations in domain-specific context. Our experiments across several benchmarks (CARTE, TextTab, and TabArena) demonstrate that CASE substantially improves the performance of tabular learners on semantically rich datasets, particularly in low-data regimes.

Related papers

• MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image [52.47434184153733]
  We introduce MulTaBench, a benchmark of 40 datasets split equally between image-tabular and text-tabular tasks.<n>We show that the gains from target-aware representation tuning generalize across both text and image modalities.
  arXiv  Detail & Related papers  (2026-05-11T14:12:05Z)
• Beyond Statistical Co-occurrence: Unlocking Intrinsic Semantics for Tabular Data Clustering [52.97486694817375]
  This paper proposes Tabular-Augmented Contrastive Clustering (TagCC) to bridge the gap between dataset-specific statistics and intrinsic semantic knowledge.<n>TagCC distills underlying data semantics into textual anchors via semantic-aware transformation.<n>It is jointly optimized with a clustering objective, ensuring that the learned representations are both semantically coherent and clustering-friendly.
  arXiv  Detail & Related papers  (2026-04-13T00:25:22Z)
• Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models [64.58262227709842]
  ARISE (Attention-weighted Representation with Integrated Semantic Embeddings) is presented.<n>It builds semantic-aware representations that complement the metric space of categorical data for accurate clustering.<n>Experiments on eight benchmark datasets demonstrate consistent improvements over seven representative counterparts.
  arXiv  Detail & Related papers  (2026-01-03T11:37:46Z)
• TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval [1.4995278030955144]
  We introduce TabGemma, a schema-agnostic in-context learner that treats rows as sequences.<n>For inference, we use a compact n-gram-based retrieval to select informative exemplars that fit within a 128k-token window.<n>For regression, it is competitive at small sample sizes but trails conventional approaches as data grows.
  arXiv  Detail & Related papers  (2025-11-05T15:51:03Z)
• ConTextTab: A Semantics-Aware Tabular In-Context Learner [1.0999592665107414]
  We introduce ConTextTab, integrating semantic understanding and alignment into a table-native ICL framework.<n>Our model is competitive with SOTA across a broad set of benchmarks while setting a new standard on the semantically rich CARTE benchmark.
  arXiv  Detail & Related papers  (2025-06-12T13:57:29Z)
• Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard [27.224577475861214]
  Tabular data, a fundamental data format in machine learning, is predominantly utilized in competitions and real-world applications.<n>We propose an in-context ensemble framework for tabular prediction that leverages large language models.<n>Our method constructs a context around each test instance using its nearest neighbors and the predictions from a pool of external models.
  arXiv  Detail & Related papers  (2025-05-19T17:52:58Z)
• Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
  The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. We present TP-BERTa, a specifically pre-trained LM for tabular data prediction. A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
  arXiv  Detail & Related papers  (2024-03-04T08:38:56Z)
• Language Models As Semantic Indexers [78.83425357657026]
  We introduce LMIndexer, a self-supervised framework to learn semantic IDs with a generative language model. We show the high quality of the learned IDs and demonstrate their effectiveness on three tasks including recommendation, product search, and document retrieval.
  arXiv  Detail & Related papers  (2023-10-11T18:56:15Z)
• Preserving Modality Structure Improves Multi-Modal Learning [64.10085674834252]
  Self-supervised learning on large-scale multi-modal datasets allows learning semantically meaningful embeddings without relying on human annotations. These methods often struggle to generalize well on out-of-domain data as they ignore the semantic structure present in modality-specific embeddings. We propose a novel Semantic-Structure-Preserving Consistency approach to improve generalizability by preserving the modality-specific relationships in the joint embedding space.
  arXiv  Detail & Related papers  (2023-08-24T20:46:48Z)
• SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning [5.5616364225463055]
  We introduce a new framework, Subsetting features of Tabular data (SubTab) In this paper, we introduce a new framework, Subsetting features of Tabular data (SubTab) We argue that reconstructing the data from the subset of its features rather than its corrupted version in an autoencoder setting can better capture its underlying representation.
  arXiv  Detail & Related papers  (2021-10-08T20:11:09Z)
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing [117.98107557103877]
  We present GraPPa, an effective pre-training approach for table semantic parsing. We construct synthetic question-pairs over high-free tables via a synchronous context-free grammar. To maintain the model's ability to represent real-world data, we also include masked language modeling.
  arXiv  Detail & Related papers  (2020-09-29T08:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.