Active In-Context Learning for Tabular Foundation Models

• URL: http://arxiv.org/abs/2603.27385v1
• Date: Sat, 28 Mar 2026 19:37:13 GMT
• Title: Active In-Context Learning for Tabular Foundation Models
• Abstract summary: We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules.<n>Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines.
• Score: 0.675253870287079
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.

Related papers

• TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention [47.480923979431886]
  Tabular foundation models perform prediction via in-context learning, inferring test labels directly from labeled training examples.<n>Recent models often improve accuracy with increasingly complex architectures, incurring higher inference cost and limiting practical deployment.<n>We show that a lightweight row-wise attention-only backbone can remain highly competitive with two simple enhancements.
  arXiv  Detail & Related papers  (2026-06-05T14:55:30Z)
• LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models [56.999481798138625]
  LimiX-2M is a 2M- parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used benchmarks.<n>We present a unified tokenize-and-route framework for strong Tabular foundation models (TFMs)<n>Results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs.
  arXiv  Detail & Related papers  (2026-06-03T06:07:33Z)
• Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment [12.754011132549167]
  DistPFN is the first test-time posterior adjustment method designed for tabular foundation models.<n>We evaluate our methods on over 250 OpenML datasets, demonstrating substantial improvements for various TabPFN-based models.
  arXiv  Detail & Related papers  (2026-05-06T00:01:47Z)
• Gradient Free Deep Reinforcement Learning With TabPFN [23.140580395364207]
  We propose TabPFN RL, a gradient free deep RL framework that repurposes the meta trained transformer TabPFN as a Q function approximator.<n>We use TabPFN to predict Q values using inference only, thereby eliminating the need for back propagation at both training and inference.<n>Our results establish prior fitted networks such as TabPFN as a viable foundation for fast and computationally efficient RL, opening new directions for gradient free RL with large pre trained transformers.
  arXiv  Detail & Related papers  (2025-09-14T13:09:58Z)
• Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Data [2.2682391370097794]
  We introduce a tiled-block strategy to compute attention within the TabPFN framework.<n>This design is compatible with standard GPU setups.<n>We demonstrate the effectiveness of our approach on the standard TabArena benchmark.
  arXiv  Detail & Related papers  (2025-08-30T02:57:01Z)
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data [15.08819125687632]
  We introduce TabICL, a tabular foundation model for classification, pretrained on synthetic datasets with up to 60K samples.<n>Across 200 classification datasets from the TALENT benchmark, TabICL is on par with TabPFNv2 while being systematically faster (up to 10 times)<n>On 53 datasets with over 10K samples, TabICL surpasses both TabPFNv2 and CatBoost, demonstrating the potential of ICL for large data.
  arXiv  Detail & Related papers  (2025-02-08T13:25:04Z)
• Tokenize features, enhancing tables: the FT-TABPFN model for tabular classification [13.481699494376809]
  FT-TabPFN is an enhanced version of TabPFN that includes a novel Feature Tokenization layer to better handle classification features. Our full source code is available for community use and development.
  arXiv  Detail & Related papers  (2024-06-11T02:13:46Z)
• TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks [90.00817095558094]
  Prior-data fitted networks (PFNs) make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. We introduce TuneTables, a parameter-efficient fine-tuning strategy for PFNs that compresses large datasets into a smaller learned context. We show that TuneTables can be used as an interpretability tool and can even be used to mitigate biases by optimizing a fairness objective.
  arXiv  Detail & Related papers  (2024-02-17T00:02:23Z)
• ProTeCt: Prompt Tuning for Taxonomic Open Set Classification [59.59442518849203]
  Few-shot adaptation methods do not fare well in the taxonomic open set (TOS) setting. We propose a prompt tuning technique that calibrates the hierarchical consistency of model predictions. A new Prompt Tuning for Hierarchical Consistency (ProTeCt) technique is then proposed to calibrate classification across label set granularities.
  arXiv  Detail & Related papers  (2023-06-04T02:55:25Z)
• Is margin all you need? An extensive empirical study of active learning on tabular data [66.18464006872345]
  We analyze the performance of a variety of active learning algorithms on 69 real-world datasets from the OpenML-CC18 benchmark. Surprisingly, we find that the classical margin sampling technique matches or outperforms all others, including current state-of-art.
  arXiv  Detail & Related papers  (2022-10-07T21:18:24Z)
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second [48.87527918630822]
  We present TabPFN, a trained Transformer that can do supervised classification for small datasets in less than a second. TabPFN performs in-context learning (ICL), it learns to make predictions using sequences of labeled examples. We show that our method clearly outperforms boosted trees and performs on par with complex state-of-the-art AutoML systems with up to 230$times$ speedup.
  arXiv  Detail & Related papers  (2022-07-05T07:17:43Z)
• Meta-Generating Deep Attentive Metric for Few-shot Classification [53.07108067253006]
  We present a novel deep metric meta-generation method to generate a specific metric for a new few-shot learning task. In this study, we structure the metric using a three-layer deep attentive network that is flexible enough to produce a discriminative metric for each task. We gain surprisingly obvious performance improvement over state-of-the-art competitors, especially in the challenging cases.
  arXiv  Detail & Related papers  (2020-12-03T02:07:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.