Accelerate Scaling of LLM Finetuning via Quantifying the Coverage and Depth of Instruction Set

• URL: http://arxiv.org/abs/2509.06463v2
• Date: Tue, 28 Oct 2025 08:59:44 GMT
• Title: Accelerate Scaling of LLM Finetuning via Quantifying the Coverage and Depth of Instruction Set
• Authors: Chengwei Wu, Li Du, Hanyu Zhao, Yiming Ju, Jiapu Wang, Tianyu Chen, Haoyi Zhou,
• Abstract summary: Scaling the amount of data used for supervied fine-tuning(SFT) does not guarantee the proportional gains in model performance.<n>This work identifies two fundamental dataset properties that govern SFT scalability.<n>We propose the textbfInformation Landscape Approximation (ILA), a model-agnostic data selection framework.
• Score: 37.26992936545316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling the amount of data used for supervied fine-tuning(SFT) does not guarantee the proportional gains in model performance, highlighting a critical need to understand what makes training samples effective. This work identifies two fundamental dataset properties that govern SFT scalability: \textbf{semantic coverage}, or the breadth of task domains, and \textbf{information depth}, or the richness of individual examples. We demonstrate that simple proxies for these properties explain the majority of validation loss variance in our experiments. In this work, we further propose the \textbf{Information Landscape Approximation (ILA)}, a model-agnostic data selection framework that jointly optimizes for these two factors. ILA constructs compact subsets that approximate the informational value of large datasets. Empirical results show that models tuned on ILA-selected data achieve faster and more sustained performance improvements across diverse tasks and model sizes compared to existing methods, a phenomenon we term \textbf{accelerated scaling}.

Related papers

• EfficientFSL: Enhancing Few-Shot Classification via Query-Only Tuning in Vision Transformers [4.880377460177786]
  Large models such as Vision Transformers (ViTs) have demonstrated remarkable superiority over smaller architectures like ResNet in few-shot classification.<n>We propose EfficientFSL, a query-only fine-tuning framework tailored specifically for few-shot classification with ViT.<n>With minimal trainable parameters, EfficientFSL achieves state-of-the-art performance on four in-domain few-shot datasets and six cross-domain datasets.
  arXiv  Detail & Related papers  (2026-01-13T12:33:02Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• Does This Look Familiar to You? Knowledge Analysis via Model Internal Representations [0.0]
  There is no clearly established methodology for effective training data selection.<n>Model Internal Representations (KAMIR) is a novel approach that overcomes these limitations.<n>It can be applied to a wide range of tasks such as machine reading comprehension and summarization.
  arXiv  Detail & Related papers  (2025-09-09T01:08:15Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• RAISE: Reinforced Adaptive Instruction Selection For Large Language Models [48.63476198469349]
  We propose a task-objective-driven instruction selection framework RAISE(Reinforced Adaptive Instruction SElection)<n> RAISE incorporates the entire instruction fine-tuning process into optimization, selecting instructions at each step based on the expected impact of each instruction on model performance improvement.<n>Experiments and result analysis prove the superiority of our method compared with other instruction selection methods.
  arXiv  Detail & Related papers  (2025-04-09T21:17:52Z)
• PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection [68.8373788348678]
  Visual instruction tuning adapts pre-trained Multimodal Large Language Models to follow human instructions.<n>PRISM is the first training-free framework for efficient visual instruction selection.<n>It reduces the end-to-end time for data selection and model tuning to just 30% of conventional pipelines.
  arXiv  Detail & Related papers  (2025-02-17T18:43:41Z)
• Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models [10.517704202614091]
  sparse Mixture-of-Experts (MoEs) allow scaling the number of parameters without proportionally increasing the FLOPs per example.<n>We investigate how varying the sparsity level, i.e., the fraction of inactive parameters, impacts model's performance during pretraining and downstream few-shot evaluation.
  arXiv  Detail & Related papers  (2025-01-21T18:51:15Z)
• Eliciting Causal Abilities in Large Language Models for Reasoning Tasks [14.512834333917414]
  We introduce the Self-Causal Instruction Enhancement (SCIE) method, which enables LLMs to generate high-quality, low-quantity observational data.<n>In SCIE, the instructions are treated as the treatment, and textual features are used to process natural language.<n>Our method effectively generates instructions that enhance reasoning performance with reduced training cost of prompts.
  arXiv  Detail & Related papers  (2024-12-19T17:03:02Z)
• A CLIP-Powered Framework for Robust and Generalizable Data Selection [51.46695086779598]
  Real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance.<n>Data selection has shown promise in identifying the most representative samples from the entire dataset.<n>We propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection.
  arXiv  Detail & Related papers  (2024-10-15T03:00:58Z)
• Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs [63.36637269634553]
  We introduce a novel approach where LLMs are fine-tuned to generate a sequence of Diverse Chains of Thought (DCoT) within a single inference step.<n>We show that fine-tuning on DCoT improves performance over the CoT baseline across model families and scales.<n>Our work is also significant because both quantitative analyses and manual evaluations reveal the observed gains stem from the models' ability to refine an initial reasoning chain.
  arXiv  Detail & Related papers  (2024-07-03T15:01:18Z)
• Tabular Data Generation Models: An In-Depth Survey and Performance Benchmarks with Extensive Tuning [2.855894241049707]
  We investigate the impact of dataset-specific tuning on five recent model families for tabular data generation through an extensive benchmark on 16 datasets.<n>Our benchmark demonstrates that, for most models, large-scale dataset-specific tuning substantially improves performance compared to the original configurations.
  arXiv  Detail & Related papers  (2024-06-18T07:27:38Z)
• LESS: Selecting Influential Data for Targeted Instruction Tuning [64.78894228923619]
  We propose LESS, an efficient algorithm to estimate data influences and perform Low-rank gradiEnt Similarity Search for instruction data selection. We show that training on a LESS-selected 5% of the data can often outperform training on the full dataset across diverse downstream tasks. Our method goes beyond surface form cues to identify data that the necessary reasoning skills for the intended downstream application.
  arXiv  Detail & Related papers  (2024-02-06T19:18:04Z)
• Instruction Position Matters in Sequence Generation with Large Language Models [67.87516654892343]
  Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization. We propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences.
  arXiv  Detail & Related papers  (2023-08-23T12:36:57Z)
• Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources [9.359395812292291]
  This paper proposes a framework called or>, which predicts model performance and supports data selection decisions based on partial samples of prospective data sources. or> significantly improves existing performance scaling approaches in terms of both accuracy of performance inference and computation costs associated with constructing the performance. Also, or> outperforms by a wide margin in data selection effectiveness compared to a range of other off-the-shelf solutions.
  arXiv  Detail & Related papers  (2023-07-05T17:33:41Z)
• Robust Prompt Optimization for Large Language Models Against Distribution Shifts [80.6757997074956]
  Large Language Model (LLM) has demonstrated significant ability in various Natural Language Processing tasks. We propose a new problem of robust prompt optimization for LLMs against distribution shifts. This problem requires the prompt optimized over the labeled source group can simultaneously generalize to an unlabeled target group.
  arXiv  Detail & Related papers  (2023-05-23T11:30:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.