OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training

• URL: http://arxiv.org/abs/2603.28858v2
• Date: Sat, 04 Apr 2026 01:16:42 GMT
• Title: OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
• Abstract summary: We propose OptiMer, which decouples ratio selection from training.<n>Experiments on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code)
• Score: 14.608696839111134
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual pre-training is widely used to adapt LLMs to target languages and domains, yet the mixture ratio of training data remains a sensitive hyperparameter that is expensive to tune: they must be fixed before training begins, and a suboptimal choice can waste weeks of compute. In this work, we propose OptiMer, which decouples ratio selection from training: we train one CPT model per dataset, extract each model's distribution vector, which represents the parameter shift induced by that dataset, and search for optimal composition weights post-hoc via Bayesian optimization. Experiments on Gemma 3 27B across languages (Japanese, Chinese) and domains (Math, Code) show that OptiMer consistently outperforms data mixture and model averaging baselines with 15-35 times lower search cost. Key findings reveal that 1) the optimized weights can be interpreted as data mixture ratios, and retraining with these ratios improves data mixture CPT, and 2) the same vector pool can be re-optimized for a given objective without any retraining, producing target-tailored models on demand. Our work establishes that data mixture ratio selection, traditionally a pre-training decision, can be reformulated as a post-hoc optimization over distribution vectors, offering a more flexible paradigm for continual pre-training.

Related papers

• FastMix: Fast Data Mixture Optimization via Gradient Descent [55.109758242985556]
  Fastmix is a novel framework that automates data mixture discovery while training only a single proxy model.<n>We show that optimizing mixture ratios is mathematically equivalent to assigning per-source loss weights under uniform source sampling.<n>FASTmix outperforms baselines while drastically reducing search cost across pre-intensive and post-training.
  arXiv  Detail & Related papers  (2026-06-12T21:46:57Z)
• DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models [51.48564522455171]
  We present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory.<n>DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment adjustment, and data reweighting.<n>It provides trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training.
  arXiv  Detail & Related papers  (2026-03-27T08:28:02Z)
• Linear Model Merging Unlocks Simple and Scalable Multimodal Data Mixture Optimization [38.78268216433473]
  We study model merging as an efficient strategy for estimating the performance of different data mixtures.<n>We conduct experiments on 14 multimodal benchmarks, and empirically demonstrate that the proxy models exhibit a high rank correlation with models trained on actual data mixtures.
  arXiv  Detail & Related papers  (2026-02-04T16:06:39Z)
• MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging [72.00014675808228]
  textbfMix determines optimal data mixing ratios by repurposing model merging weights as a high-fidelity, low-cost performance proxy.<n>Experiments on models with 8B and 16B parameters validate that MergeMix achieves performance comparable to or surpassing exhaustive manual tuning.
  arXiv  Detail & Related papers  (2026-01-25T14:31:57Z)
• Optimizing the Training Diet: Data Mixture Search for Robust Time Series Forecasting [0.8665758002017515]
  We show that, in some cases, "less is more" when considering datasets.<n>We introduce a framework for discovering the optimal "training diet" from a large, unlabeled time series corpus.
  arXiv  Detail & Related papers  (2025-12-12T13:26:07Z)
• Composable Cross-prompt Essay Scoring by Merging Models [7.5702468122067685]
  Cross-prompt automated essay scoring typically trains models jointly on all source prompts.<n>We propose a source-free adaptation approach that selectively merges individually trained source models' parameters instead of datasets.
  arXiv  Detail & Related papers  (2025-05-24T06:28:21Z)
• Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models [24.396525123797073]
  We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture.<n>We use this approximation as a source of additional features in a regression model, trained from observations of model loss for a small number of mixtures.
  arXiv  Detail & Related papers  (2025-02-21T21:27:48Z)
• AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs [59.12061830645018]
  We show that data mixtures that perform well at smaller scales may not retain their advantage at larger scales.<n>We propose AutoScale, a two-stage, scale-aware data composition framework.
  arXiv  Detail & Related papers  (2024-07-29T17:06:30Z)
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance [55.872926690722714]
  We study the predictability of model performance regarding the mixture proportions in function forms.<n>We propose nested use of the scaling laws of training steps, model sizes, and our data mixing law.<n>Our method effectively optimize the training mixture of a 1B model trained for 100B tokens in RedPajama.
  arXiv  Detail & Related papers  (2024-03-25T17:14:00Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• Efficient Online Data Mixing For Language Model Pre-Training [101.45242332613944]
  Existing data selection methods suffer from slow and computationally expensive processes. Data mixing, on the other hand, reduces the complexity of data selection by grouping data points together. We develop an efficient algorithm for Online Data Mixing (ODM) that combines elements from both data selection and data mixing.
  arXiv  Detail & Related papers  (2023-12-05T00:42:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.