Scaling Laws for Optimal Data Mixtures

• URL: http://arxiv.org/abs/2507.09404v1
• Date: Sat, 12 Jul 2025 21:16:08 GMT
• Title: Scaling Laws for Optimal Data Mixtures
• Authors: Mustafa Shukor, Louis Bethune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, Pierre Ablin,
• Abstract summary: We propose a systematic method to determine the optimal data mixture for any target domain using scaling laws.<n>We validate the universality of these scaling laws by demonstrating their predictive power in three distinct and large-scale settings.
• Score: 30.981047302765138
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance. The standard approach to selecting this mixture relies on trial and error, which becomes impractical for large-scale pretraining. We propose a systematic method to determine the optimal data mixture for any target domain using scaling laws. Our approach accurately predicts the loss of a model of size $N$ trained with $D$ tokens and a specific domain weight vector $h$. We validate the universality of these scaling laws by demonstrating their predictive power in three distinct and large-scale settings: large language model (LLM), native multimodal model (NMM), and large vision models (LVM) pretraining. We further show that these scaling laws can extrapolate to new data mixtures and across scales: their parameters can be accurately estimated using a few small-scale training runs, and used to estimate the performance at larger scales and unseen domain weights. The scaling laws allow to derive the optimal domain weights for any target domain under a given training budget ($N$,$D$), providing a principled alternative to costly trial-and-error methods.

Related papers

• DataDecide: How to Predict Best Pretraining Data with Small Experiments [67.95896457895404]
  We release models, data, and evaluations in DataDecide -- the most extensive open suite of models over differences in data and scale.<n>We conduct controlled pretraining experiments across 25 corpora with differing sources, deduplication, and filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random seeds.
  arXiv  Detail & Related papers  (2025-04-15T17:02:15Z)
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining [59.369484219304866]
  In this study, we conduct an unprecedented empirical investigationtext- training over 3,700 Large Language Models (LLMs) from scratch across 100 trillion tokens.<n>We empirically observe that, under fixed model size ($N$) and dataset size ($D$), the hyperparameter landscape exhibits convexity with a broad optimum.<n>Building on this insight, we formally define and empirically validate the Step Law: The optimal learning rate follows a power-law relationship with $N$ and $D$, while the optimal batch size is primarily influenced by $D$ and remains largely invariant to $N$.
  arXiv  Detail & Related papers  (2025-03-06T18:58:29Z)
• The interplay between domain specialization and model size [8.653321928148547]
  We investigate the interplay between domain and model size during continued pretraining under compute-constrained scenarios.<n>Our goal is to identify an optimal training regime for this scenario and detect patterns in this interplay that can be generalized across different model sizes and domains.
  arXiv  Detail & Related papers  (2025-01-03T19:28:53Z)
• A Hitchhiker's Guide to Scaling Law Estimation [56.06982415792523]
  Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets.<n>We estimate more than 1000 scaling laws, then derive a set of best practices for estimating scaling laws in new model families.
  arXiv  Detail & Related papers  (2024-10-15T17:59:10Z)
• AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs [61.13296177652599]
  We show that data mixtures that perform well at smaller scales may not retain their advantage at larger scales.<n>We propose AutoScale, a two-stage, scale-aware data composition framework.
  arXiv  Detail & Related papers  (2024-07-29T17:06:30Z)
• More Compute Is What You Need [3.184416958830696]
  We propose a new scaling law that suggests model performance depends mostly on the amount of compute spent for transformer-based models. We predict that (a) for inference efficiency, training should prioritize smaller model sizes and larger training datasets, and (b) assuming the exhaustion of available web datasets, scaling the model size might be the only way to further improve model performance.
  arXiv  Detail & Related papers  (2024-04-30T12:05:48Z)
• Scaling Laws for Fine-Grained Mixture of Experts [4.412803924115907]
  Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. We establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity.
  arXiv  Detail & Related papers  (2024-02-12T18:33:47Z)
• DoGE: Domain Reweighting with Generalization Estimation [42.32000165235568]
  We propose DOmain reweighting with Generalization Estimation (DoGE) In our experiments, we extensively show how DoGE improves the generalization of the base model to any target data mixture. DoGE can effectively identify inter-domain dependencies, and consistently achieves better test perplexity on the target domain.
  arXiv  Detail & Related papers  (2023-10-23T22:51:58Z)
• DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining [148.90031913522648]
  We propose Domain Reweighting with Minimax Optimization (DoReMi) DoReMi first trains a small proxy model using group distributionally robust optimization (Group DRO) over domains to produce domain weights. We then resample a dataset with these domain weights and train a larger, full-sized model.
  arXiv  Detail & Related papers  (2023-05-17T17:58:13Z)
• nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales [65.01417261415833]
  We present an approach to predict the pre-training loss based on our observations that Maximal Update Parametrization (muP) enables accurate fitting of scaling laws. With around 14% of the one-time pre-training cost, we can accurately forecast the loss for models up to 52B. Our goal with nanoLM is to empower researchers with limited resources to reach meaningful conclusions on large models.
  arXiv  Detail & Related papers  (2023-04-14T00:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.