A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets

• URL: http://arxiv.org/abs/2505.06150v2
• Date: Mon, 02 Jun 2025 18:33:23 GMT
• Title: A Scaling Law for Token Efficiency in LLM Fine-Tuning Under Fixed Compute Budgets
• Authors: Ryan Lagasse, Aidan Kierans, Avijit Ghosh, Shiri Dori-Hacohen,
• Abstract summary: We introduce a scaling law for fine-tuning large language models (LLMs) under fixed compute budgets.<n>Our formulation is tuned following established procedures.
• Score: 2.619545850602691
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a scaling law for fine-tuning large language models (LLMs) under fixed compute budgets that explicitly accounts for data composition. Conventional approaches measure training data solely by total tokens, yet the number of examples and their average token length -- what we term \emph{dataset volume} -- play a decisive role in model performance. Our formulation is tuned following established procedures. Experiments on the BRICC dataset \cite{salavati2024reducing} and subsets of the MMLU dataset \cite{hendrycks2021measuringmassivemultitasklanguage}, evaluated under multiple subsampling strategies, reveal that data composition significantly affects token efficiency. These results motivate refined scaling laws for practical LLM fine-tuning in resource-constrained settings.

Related papers

• Large Language Models are Demonstration Pre-Selectors for Themselves [57.101804269100185]
  In-context learning (ICL) with large language models (LLMs) delivers strong few-shot performance by choosing few-shot demonstrations from the entire training data.<n>FEw yet Essential Demonstration prE-selectoR is a novel pre-selection framework that identifies a representative subset of demonstrations.<n>FEw yet Essential Demonstration prE-selectoR can reduce training data size by over 20% while maintaining performance.
  arXiv  Detail & Related papers  (2025-06-06T12:29:03Z)
• FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain [14.109309236798518]
  Supervised fine-tuning (SFT) is a standard approach to adapting large language models (LLMs) to new domains.<n>In this work, we improve the statistical efficiency of SFT by selecting an informative subset of training examples.
  arXiv  Detail & Related papers  (2025-05-20T18:41:34Z)
• Rethinking Data: Towards Better Performing Domain-Specific Small Language Models [0.0]
  This paper presents our approach to finetuning a small Language Models (LM)<n>We achieve this by improving data quality at each stage of the LM training pipeline.<n>We improve the model generalization ability by merging the models fine-tuned with different parameters on different data subsets.
  arXiv  Detail & Related papers  (2025-03-03T12:19:12Z)
• LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws [21.053622641336744]
  Loss-to-loss scaling laws relate losses across pretraining datasets and downstream tasks.<n>Our experiments reveal that the pretraining data and tokenizer determine the scaling trend.
  arXiv  Detail & Related papers  (2025-02-17T18:45:25Z)
• Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models [15.603556124006479]
  We propose retrieval-augmented language models for scalable TabICL.<n>Our approach incorporates a customized retrieval module, combined with retrieval-guided instruction-tuning for LLMs.<n>This enables LLMs to effectively leverage larger datasets, achieving significantly improved performance across 69 widely recognized datasets.
  arXiv  Detail & Related papers  (2025-02-05T13:16:41Z)
• Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning [61.99353167168545]
  We show that fine-tuning with LLM-generated data improves target task performance and reduces non-target task degradation.<n>This is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2025-01-24T08:18:56Z)
• Optimizing Pretraining Data Mixtures with LLM-Estimated Utility [52.08428597962423]
  Large Language Models improve with increasing amounts of high-quality training data.<n>We find token-counts outperform manual and learned mixes, indicating that simple approaches for dataset size and diversity are surprisingly effective.<n>We propose two complementary approaches: UtiliMax, which extends token-based $200s by incorporating utility estimates from reduced-scale ablations, achieving up to a 10.6x speedup over manual baselines; and Model Estimated Data Utility (MEDU), which leverages LLMs to estimate data utility from small samples, matching ablation-based performance while reducing computational requirements by $simx.
  arXiv  Detail & Related papers  (2025-01-20T21:10:22Z)
• Aligning Instruction Tuning with Pre-training [81.4748965653345]
  We propose Aligning Instruction Tuning with Pre-training (AITP) to align instruction tuning with pre-training distributions.<n>We show consistent performance improvements with AITP on three fully open large language models (LLMs) across eight benchmarks.
  arXiv  Detail & Related papers  (2025-01-16T08:27:40Z)
• Entropy Law: The Story Behind Data Compression and LLM Performance [115.70395740286422]
  We find that model performance is negatively correlated to the compression ratio of training data, which usually yields a lower training loss. Based on the findings of the entropy law, we propose a quite efficient and universal data selection method. We also present an interesting application of entropy law that can detect potential performance risks at the beginning of model training.
  arXiv  Detail & Related papers  (2024-07-09T08:14:29Z)
• Through the Thicket: A Study of Number-Oriented LLMs derived from Random Forest Models [0.0]
  Large Language Models (LLMs) have shown exceptional performance in text processing. This paper proposes a novel approach to training LLMs using knowledge transfer from a random forest (RF) ensemble. We generate outputs for fine-tuning, enhancing the model's ability to classify and explain its decisions.
  arXiv  Detail & Related papers  (2024-06-07T13:31:51Z)
• A Context-Aware Approach for Enhancing Data Imputation with Pre-trained Language Models [0.18416014644193068]
  CRILM uses pre-trained language models to create contextually relevant descriptors for missing values.<n>Our evaluations demonstrate CRILM's superior performance and robustness across MCAR, MAR, and challenging MNAR scenarios.
  arXiv  Detail & Related papers  (2024-05-28T00:08:29Z)
• Amortizing intractable inference in large language models [56.92471123778389]
  We use amortized Bayesian inference to sample from intractable posterior distributions. We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem.
  arXiv  Detail & Related papers  (2023-10-06T16:36:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.