Exploring Fine-Tuning for Tabular Foundation Models

• URL: http://arxiv.org/abs/2601.09654v1
• Date: Wed, 14 Jan 2026 17:40:46 GMT
• Title: Exploring Fine-Tuning for Tabular Foundation Models
• Authors: Aditya Tanna, Pratinav Seth, Mohamed Bouadi, Vinay Kumar Sankarapu,
• Abstract summary: This work presents the first comprehensive study of fine-tuning in Tabular Foundation Models (TFMs)<n>We compare Zero-Shot, Meta-Learning, Supervised (SFT), and parameter-efficient (PEFT) approaches, analyzing how dataset factors such as imbalance, size, and dimensionality affect outcomes.<n>Our findings cover performance, calibration, and fairness, offering practical guidelines on when fine-tuning is most beneficial and its limitations.
• Score: 3.884856136722027
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tabular Foundation Models (TFMs) have recently shown strong in-context learning capabilities on structured data, achieving zero-shot performance comparable to traditional machine learning methods. We find that zero-shot TFMs already achieve strong performance, while the benefits of fine-tuning are highly model and data-dependent. Meta-learning and PEFT provide moderate gains under specific conditions, whereas full supervised fine-tuning (SFT) often reduces accuracy or calibration quality. This work presents the first comprehensive study of fine-tuning in TFMs across benchmarks including TALENT, OpenML-CC18, and TabZilla. We compare Zero-Shot, Meta-Learning, Supervised (SFT), and parameter-efficient (PEFT) approaches, analyzing how dataset factors such as imbalance, size, and dimensionality affect outcomes. Our findings cover performance, calibration, and fairness, offering practical guidelines on when fine-tuning is most beneficial and its limitations.

Related papers

• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels [72.3670919950349]
  Large language models (LLMs) acquire substantial world knowledge during pre-training.<n>Post-training techniques such as supervised fine-tuning (SFT) shape this knowledge change behavior.<n>We evaluate closed-book question answering (CBQA) performance across five LLMs from the LLaMA-2 and LLaMA-3 families.
  arXiv  Detail & Related papers  (2025-09-20T09:40:32Z)
• LoFT: Parameter-Efficient Fine-Tuning for Long-tailed Semi-Supervised Learning in Open-World Scenarios [19.673195747304195]
  We propose a novel framework for long-tailed semi-supervised learning: LoFT (Long-tailed semi-supervised learning via parameter-efficient Fine-Tuning)<n>We show that fine-tuned foundation models can generate more reliable pseudolabels, thereby benefiting imbalanced learning.<n>We also explore a more practical setting by investigating semi-supervised learning under open-world conditions.
  arXiv  Detail & Related papers  (2025-09-12T02:28:32Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality [22.105116314320696]
  Supervised fine-tuning (SFT) is a critical step in aligning large language models with human instructions and values.<n>We trained a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks.<n>We then identified the dataset properties that matter most and examined the layer-wise modifications introduced by SFT.
  arXiv  Detail & Related papers  (2025-06-17T16:13:15Z)
• LIFT+: Lightweight Fine-Tuning for Long-Tail Learning [45.187004699024435]
  LIFT+ is an innovative lightweight fine-tuning framework to optimize consistent class conditions.<n>Our framework provides an efficient and accurate pipeline that facilitates fast convergence and model compactness.
  arXiv  Detail & Related papers  (2025-04-17T18:50:47Z)
• SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning [8.566687323389264]
  We introduce Sparse Adaptation for Fine-Tuning (SAFT), a method that prevents fine-tuning from forgetting the general knowledge in the pre-trained model. With only 0.1% of the model parameters, SAFT can significantly improve the performance of CLIP. On the few-shot learning benchmark of ImageNet, SAFT gives a gain of 5.15% on average over the conventional fine-tuning method in OOD settings.
  arXiv  Detail & Related papers  (2024-07-03T11:56:55Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Comparative Analysis of Different Efficient Fine Tuning Methods of Large Language Models (LLMs) in Low-Resource Setting [0.0]
  We try to push the understanding of different fine-tuning strategies for large language models (LLMs) We compare state-of-the-art methods like vanilla fine-tuning and Pattern-Based Fine-Tuning (PBFT) on pre-trained models across two datasets, COLA and MNLI. Our findings suggest that these alternative strategies can exhibit out-of-domain generalization comparable to that of vanilla FT and PBFT.
  arXiv  Detail & Related papers  (2024-05-21T20:08:52Z)
• AutoFT: Learning an Objective for Robust Fine-Tuning [60.641186718253735]
  Foundation models encode rich representations that can be adapted to downstream tasks by fine-tuning. Current approaches to robust fine-tuning use hand-crafted regularization techniques. We propose AutoFT, a data-driven approach for robust fine-tuning.
  arXiv  Detail & Related papers  (2024-01-18T18:58:49Z)
• DavIR: Data Selection via Implicit Reward for Large Language Models [62.59514469369608]
  DavIR is a model-based data selection method for post-training Large Language Models.<n>We show that 6% of Alpaca dataset selected with DavIR can steer both the LLaMA and Gemma model family to produce superior performance compared to the same models trained on the full 52K dataset.
  arXiv  Detail & Related papers  (2023-10-16T07:26:24Z)
• Strong Baselines for Parameter Efficient Few-Shot Fine-tuning [50.83426196335385]
  Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pre-training (or meta-training) phase. Recent works have shown that simply fine-tuning a pre-trained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters.
  arXiv  Detail & Related papers  (2023-04-04T16:14:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.