TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention

• URL: http://arxiv.org/abs/2606.07345v1
• Date: Fri, 05 Jun 2026 14:55:30 GMT
• Title: TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention
• Abstract summary: Tabular foundation models perform prediction via in-context learning, inferring test labels directly from labeled training examples.<n>Recent models often improve accuracy with increasingly complex architectures, incurring higher inference cost and limiting practical deployment.<n>We show that a lightweight row-wise attention-only backbone can remain highly competitive with two simple enhancements.
• Score: 47.480923979431886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular foundation models, exemplified by TabPFN, perform prediction via in-context learning, inferring test labels directly from labeled training examples. They have demonstrated competitive performance, particularly on small-to-medium datasets. However, recent tabular foundation models often improve accuracy with increasingly complex architectures, incurring higher inference cost and limiting practical deployment. In this work, we revisit the original TabPFN design and show that a lightweight row-wise attention-only backbone can remain highly competitive with two simple enhancements: a gated attention stabilization mechanism and a small set of learnable register tokens that provide global context and improve pretraining quality. The resulting model, TabSwift, supports both classification and regression, and is competitive with stronger tabular foundation models (e.g., TabPFN v2 and TabICL) while being more efficient at inference. For latency-sensitive serving, we further introduce an adaptive layer-wise early-exit mechanism that dynamically adjusts inference depth per sample. Overall, TabSwift enables efficient and anytime tabular in-context learning for practical deployments.

Related papers

• TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction [8.596879252293617]
  Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity.<n>We present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on Tab-Lite, CC18, and CTR23, at orders of magnitude faster.
  arXiv  Detail & Related papers  (2026-08-02T17:33:30Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image [52.47434184153733]
  We introduce MulTaBench, a benchmark of 40 datasets split equally between image-tabular and text-tabular tasks.<n>We show that the gains from target-aware representation tuning generalize across both text and image modalities.
  arXiv  Detail & Related papers  (2026-05-11T14:12:05Z)
• Self-Improving Tabular Language Models via Iterative Group Alignment [76.43372236405664]
  We introduce TabGRAA, the first self-improving framework for data generation via automated feedback.<n>It partitions newly generated samples into high- and low-quality groups, then prioritizes a group-relative advantage objective.<n>Experiments show TabGRAA outperforms existing methods in fidelity, utility, and privacy.
  arXiv  Detail & Related papers  (2026-04-21T01:29:52Z)
• RDBLearn: Simple In-Context Prediction Over Relational Databases [21.996337463952255]
  We show that a simple recipe can be extended to relational prediction with a simple recipe.<n>We package this approach in textitRDBLearn, an easy-to-use toolkit with a scikit-learn-style estimator interface.<n>Across a broad collection of RelBench and 4DBInfer datasets, RDBLearn is the best-performing foundation model approach we evaluate.
  arXiv  Detail & Related papers  (2026-02-14T09:24:04Z)
• TabICLv2: A better, faster, scalable, and open tabular foundation model [18.594859017648346]
  We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars.<n>Tabiclv2 generalizes effectively to million-scale datasets under 50GB GPU memory while being markedly faster than RealTabPFN-2.5.
  arXiv  Detail & Related papers  (2026-02-11T18:51:02Z)
• Comparing Task-Agnostic Embedding Models for Tabular Data [1.6479389738270018]
  This work specifically focuses on representation learning, i.e., on transferable, task-agnostic embeddings.<n>Tableizer features achieve comparable or superior performance while being up to three orders of magnitude faster than recent foundation models.
  arXiv  Detail & Related papers  (2025-11-18T09:10:40Z)
• nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN [78.62756717376563]
  nanoTabPFN is a lightweight implementation of the TabPFN v2 architecture and a corresponding training loop.<n>It achieves a performance comparable to traditional machine learning baselines within one minute of pre-training on a single GPU.
  arXiv  Detail & Related papers  (2025-11-05T16:52:51Z)
• A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities [51.08999772842298]
  Tabular Prior-data Fitted Network v2 (TabPFN v2) achieves unprecedented in-context learning performance across diverse downstream datasets.<n>We show that TabPFN v2 can infer attribute relationships even when provided with randomized attribute token inputs.<n>We demonstrate that TabPFN v2's limitations can be addressed through a test-time divide-and-context strategy.
  arXiv  Detail & Related papers  (2025-02-24T17:38:42Z)
• TabDPT: Scaling Tabular Foundation Models on Real Data [20.00390825519329]
  We propose an approach to combine ICL-based retrieval with self supervised learning to train foundation models.<n>We show that incorporating real data during the pre-training phase can lead to significantly faster training and better generalization to unseen data.<n>Our resulting model, TabDPT, achieves top performance on both regression (CTR23) and classification (CC18) benchmarks.
  arXiv  Detail & Related papers  (2024-10-23T18:00:00Z)
• Tokenize features, enhancing tables: the FT-TABPFN model for tabular classification [13.481699494376809]
  FT-TabPFN is an enhanced version of TabPFN that includes a novel Feature Tokenization layer to better handle classification features. Our full source code is available for community use and development.
  arXiv  Detail & Related papers  (2024-06-11T02:13:46Z)
• TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks [90.00817095558094]
  Prior-data fitted networks (PFNs) make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. We introduce TuneTables, a parameter-efficient fine-tuning strategy for PFNs that compresses large datasets into a smaller learned context. We show that TuneTables can be used as an interpretability tool and can even be used to mitigate biases by optimizing a fairness objective.
  arXiv  Detail & Related papers  (2024-02-17T00:02:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.