BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification

• URL: http://arxiv.org/abs/2605.06117v2
• Date: Mon, 11 May 2026 01:17:23 GMT
• Title: BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification
• Abstract summary: BoostLLM is a framework that transforms parameter-efficient fine-tuning into a multi-round residual optimization process.<n>We show that BoostLLM achieves consistent improvements over standard fine-tuning across multiple LLM backbones and datasets.<n>These results suggest that boosting can serve as a general training principle for LLM fine-tuning, particularly in low-data regimes for structured data.
• Score: 1.5002600992095827
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have recently been adapted to tabular prediction by serializing structured features into natural language, but their performance in low-data regimes remains limited compared to gradient-boosted decision trees (GBDTs). In this work, we revisit the boosting paradigm, traditionally associated with tree ensembles, and ask whether it can be applied as a general training principle for LLM fine-tuning. We propose BoostLLM, a framework that transforms parameter-efficient fine-tuning into a multi-round residual optimization process by training sequential PEFT adapters as weak learners. To incorporate tabular inductive bias, BoostLLM integrates decision-tree paths as a second input view alongside raw features; analysis reveals that the path view acts as a structured teacher in early training steps before the model shifts toward feature-driven representations. Empirically, BoostLLM achieves consistent improvements over standard fine-tuning across multiple LLM backbones and datasets, matching or surpassing XGBoost across a wide range of shot counts and outperforming GPT-4o-based methods with a 4B model. We further show that the framework scales: pairing with stronger tree models and extended boosting horizons yields additional gains under appropriate stabilization. These results suggest that boosting can serve as a general training principle for LLM fine-tuning, particularly in low-data regimes for structured data.

Related papers

• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• Boost Like a (Var)Pro: Trust-Region Gradient Boosting via Variable Projection [0.0]
  textttVPBoost fuses variable projection, a training paradigm for separable models, with a second-order weak learning strategy.<n>VPBoost learns an ensemble with improved evaluation metrics in comparison to gradient-descent-based boosting.
  arXiv  Detail & Related papers  (2026-03-24T19:01:07Z)
• Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMs [60.68927774057402]
  We show, for the first time, that a lower simplicity bias induces a better generalization.<n>Motivated by this insight, we demonstrate that the training data distribution by upsampling or augmenting examples learned later in training similarly reduces SB and leads to improved generalization.<n>Our strategy improves the performance of multiple language models including Phi2-2.7B, Llama3.2-1B, Gemma3-1B-PT, Qwen3-0.6B-Base-achieving relative accuracy gains up to 18% when fine-tuned with AdamW and Muon.
  arXiv  Detail & Related papers  (2026-01-31T07:40:36Z)
• LLMBoost: Make Large Language Models Stronger with Boosting [45.63712975323876]
  We introduce LLMBoost, a novel ensemble fine-tuning framework for ensemble learning of LLMs.<n>Inspired by the boosting paradigm, LLMBoost incorporates three key innovations. First, a cross-model attention mechanism enables successor models to access and fuse hidden states from predecessors.<n>Second, a chain training paradigm progressively fine-tunes connected models with an error-suppression objective, ensuring that each model rectifies the mispredictions of its predecessor.<n>Third, a near-parallel inference paradigm design pipelines hidden states across models layer by layer, achieving inference efficiency approaching single-model decoding.
  arXiv  Detail & Related papers  (2025-12-26T07:16:41Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Shadow-FT: Tuning Instruct Model via Training on Paired Base Model [67.20706292627106]
  Large language models (LLMs) consistently benefit from further fine-tuning on various tasks.<n>We propose a novel Shadow-FT framework to tune the Instruct models by leveraging the corresponding Base models.<n>Our proposed Shadow-FT introduces no additional parameters, is easy to implement, and significantly improves performance.
  arXiv  Detail & Related papers  (2025-05-19T05:16:21Z)
• Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping [53.454408491386886]
  bootstrapping self-alignment markedly surpasses the single-round approach. We propose Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced few-shot ability to boost zero or one-shot performance. Based on easy-to-hard training recipe, we propose SOFT+ which further boost self-alignment's performance.
  arXiv  Detail & Related papers  (2024-02-12T12:30:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.