Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining

• URL: http://arxiv.org/abs/2608.23922v1
• Date: Mon, 24 Aug 2026 23:56:38 GMT
• Title: Data Mixing as Mixture Experiment: Response Surface Methodology and Optimal Design for Large Language Model Pretraining
• Abstract summary: We show the structure of a classical mixture experiment using proxy-based methods.<n>Data domains are mixture components, token shares are component proportions, proxy-training runs are experimental design points, and validation loss defines a response surface over the probability simplex.<n>Using RegMix as an empirical case study, we demonstrate how the framework can both interpret observed mixture responses and design more efficient proxy experiments.
• Score: 1.0312968200748118
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data mixing is a central design problem in large language model pretraining: given a fixed token budget, practitioners must decide how much data to allocate to each domain. Recent proxy-based methods address this problem by training small models on candidate mixtures, fitting a response model, and using the response to select mixtures for larger-scale training. We show that this workflow has the structure of a classical mixture experiment. Under this view, data domains are mixture components, token shares are component proportions, proxy-training runs are experimental design points, and validation loss defines a response surface over the probability simplex. We develop this formulation using sparse second-order Scheffé response-surface models and construct model-robust $\mathcal{I}$-optimal designs for proxy data-mixing experiments. Using RegMix as an empirical case study, we demonstrate how the framework can both interpret observed mixture responses and design more efficient proxy experiments. The Scheffé analysis shows that domain value is strongly relational: several domains that are weak under additive effects become favourable through pairwise interactions, especially through combinations with web-derived text. The sparse Scheffé model preserves mixture rankings across model scales and remains competitive with a flexible machine-learning predictor while providing an explicit decomposition of additive and interaction effects. In a simulation study calibrated to observed proxy-training responses, model-robust $\mathcal{I}$-optimal designs recover the relevant mixture ordering after removing about 25\% of the original proxy runs. These results suggest that LLM data mixing should be treated not only as a prediction problem, but also as an experimental-design problem in which the proxy mixtures themselves can be chosen to improve statistical efficiency.

Related papers

• CausalMix: Data Mixture as Causal Inference for Language Model Training [25.091228095854415]
  In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance.<n>Recent methods optimize mixture weights via proxy models, but they rely on the assumption of static data distributions.<n>We propose CausalMix to address this limitation by casting data mixture optimization as a causal inference problem.
  arXiv  Detail & Related papers  (2026-07-01T15:56:11Z)
• FastMix: Fast Data Mixture Optimization via Gradient Descent [55.109758242985556]
  Fastmix is a novel framework that automates data mixture discovery while training only a single proxy model.<n>We show that optimizing mixture ratios is mathematically equivalent to assigning per-source loss weights under uniform source sampling.<n>FASTmix outperforms baselines while drastically reducing search cost across pre-intensive and post-training.
  arXiv  Detail & Related papers  (2026-06-12T21:46:57Z)
• Data Mixing for Large Language Models Pretraining: A Survey and Outlook [52.66761353708619]
  Large language models (LLMs) rely on pretraining on massive and heterogeneous corpora.<n>Unlike sample-level data selection, data mixing optimize domain-level sampling weights to allocate limited budgets more effectively.<n>This paper provides a comprehensive review of data mixing for LLM pretraining.
  arXiv  Detail & Related papers  (2026-03-25T13:30:40Z)
• Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training [16.022416196267937]
  We propose Decouple Searching from Training Mix (DeMix), a novel framework that leverages model merging to predict optimal data ratios.<n>We show that DeMix breaks the trade-off between sufficiency, accuracy and efficiency, obtaining the optimal mixture with higher benchmark performance at lower search cost.
  arXiv  Detail & Related papers  (2026-01-31T14:27:46Z)
• Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models [24.396525123797073]
  We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture.<n>We use this approximation as a source of additional features in a regression model, trained from observations of model loss for a small number of mixtures.
  arXiv  Detail & Related papers  (2025-02-21T21:27:48Z)
• Amortized Bayesian Mixture Models [1.3976439685325095]
  This paper introduces a novel extension of Amortized Bayesian Inference (ABI) tailored to mixture models.<n>We factorize the posterior into a distribution of the parameters and a distribution of (categorical) mixture indicators, which allows us to use a combination of generative neural networks.<n>The proposed framework accommodates both independent and dependent mixture models, enabling filtering and smoothing.
  arXiv  Detail & Related papers  (2025-01-17T14:51:03Z)
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance [55.872926690722714]
  We study the predictability of model performance regarding the mixture proportions in function forms.<n>We propose nested use of the scaling laws of training steps, model sizes, and our data mixing law.<n>Our method effectively optimize the training mixture of a 1B model trained for 100B tokens in RedPajama.
  arXiv  Detail & Related papers  (2024-03-25T17:14:00Z)
• Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
  Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers. We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes. We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
  arXiv  Detail & Related papers  (2023-08-28T18:48:34Z)
• Harnessing Hard Mixed Samples with Decoupled Regularizer [69.98746081734441]
  Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. In this paper, we propose an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM) DM can adaptively utilize hard mixed samples to mine discriminative features without losing the original smoothness of mixup.
  arXiv  Detail & Related papers  (2022-03-21T07:12:18Z)
• Robust Finite Mixture Regression for Heterogeneous Targets [70.19798470463378]
  We propose an FMR model that finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously. We provide non-asymptotic oracle performance bounds for our model under a high-dimensional learning framework. The results show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-12T03:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.