Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model

• URL: http://arxiv.org/abs/2608.13277v1
• Date: Thu, 13 Aug 2026 14:13:46 GMT
• Title: Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
• Abstract summary: Mixture of Training (MoT) is a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks.<n>MoT trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass.<n>We present MoT as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.
• Score: 1.8438975114331797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixture of Training (MoT), a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks, trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass. On a 1.3B-parameter Gemma-style model trained on C4, MoT provides a small-scale proof of mechanism: independently trained depth slices can be recomposed into a usable language model, and a quality-parity schedule reaches the same reported perplexity as the monolithic baseline. This parity setting processes more aggregate tokens and has a shorter idealized layer-equivalent critical path after aligner preparation; its effective compute advantage depends on reusing the aligner across runs. We therefore present MoT not as a general replacement for monolithic pre-training, but as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.

Related papers

• Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs [36.805962131025694]
  We introduce ADAPT---Amortized Distillation Across PostTrained LLMs---a framework for amortizing distillation across both axes of a model family.<n>First, a two-phase distillation procedure constructs post-trained students through pre-training alignment and supervised fine-tuning distillation.<n>The resulting continuum of models also enables adaptive model-size selection at inference time.
  arXiv  Detail & Related papers  (2026-08-24T06:36:32Z)
• Miles: Metric Learning with Expandable Subspace for Pre-Trained Model-Based Class-Incremental Learning [57.237835833636154]
  We propose MetrIc Learning with Expandable Subspace (Miles) to harness the prior information within pre-trained knowledge.<n>Miles achieves state-of-the-art performance in various CIL settings.
  arXiv  Detail & Related papers  (2026-07-20T06:22:02Z)
• Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts [76.12264847243816]
  We present BAR, which trains independent domain experts, each through its own mid-training, supervised finetuning, and reinforcement learning pipeline, and composes them via a Mixture-of-Experts architecture with lightweight router training.<n>At the 7B scale, with experts for math, code, tool use, and safety, BAR achieves an overall score of 49.1.<n>We show that modular training provides a structural advantage: by isolating each domain, it avoids the catastrophic forgetting that occurs when late-stage RL degrades capabilities from earlier training stages, while significantly reducing the cost and complexity of updating or adding a domain.
  arXiv  Detail & Related papers  (2026-04-20T16:24:41Z)
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs [80.72350166388601]
  Nemotron Elastic is a framework for building reasoning-oriented LLMs.<n>It embeds nested submodels within a single parent model.<n>Each of these submodels shares weights with the parent model and can be extracted zero-shot during deployment.
  arXiv  Detail & Related papers  (2025-11-20T18:59:21Z)
• Efficient Regression-Based Training of Normalizing Flows for Boltzmann Generators [85.25962679349551]
  Boltzmann Generators (BGs) offer efficient sampling and likelihoods, but their training via maximum likelihood is often unstable and computationally challenging.<n>We propose Regression Training of Normalizing Flows (RegFlow), a novel and scalable-based training objective that bypasses the numerical instability and computational challenge of conventional maximum likelihood training.
  arXiv  Detail & Related papers  (2025-06-01T20:32:27Z)
• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network)<n>After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference.<n>We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• Tuning Language Models by Mixture-of-Depths Ensemble [23.10522891268232]
  Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for training and final-layer representations for predictions. We find that focusing training efforts on intermediate layers can yield training losses comparable to those of final layers. We introduce a novel tuning framework, Mixture-of-Depths (MoD), which trains late layers as ensembles contributing to the final logits.
  arXiv  Detail & Related papers  (2024-10-16T22:51:45Z)
• Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Training [17.885679388567887]
  ProFL is a new framework that effectively addresses the memory constraints in FL.<n>It partitions the model into blocks based on its original architecture and trains each block in a progressive fashion.<n>ProFL effectively reduces the peak memory footprint by up to 57.4% and improves model accuracy by up to 82.4%.
  arXiv  Detail & Related papers  (2024-04-20T11:08:07Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• MILO: Model-Agnostic Subset Selection Framework for Efficient Model Training and Tuning [68.12870241637636]
  We propose MILO, a model-agnostic subset selection framework that decouples the subset selection from model training. Our empirical results indicate that MILO can train models $3times - 10 times$ faster and tune hyperparameters $20times - 75 times$ faster than full-dataset training or tuning without performance.
  arXiv  Detail & Related papers  (2023-01-30T20:59:30Z)
• Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training [36.5936227129021]
  It is possible to expand pretrained Masked Language Models to new languages by learning a new set of embeddings, while keeping the transformer body frozen. We propose mini-model adaptation, a compute-efficient alternative that builds a shallow mini-model from a fraction of a large model's parameters. New language-specific embeddings can then be efficiently trained over the mini-model and plugged into the aligned large model for rapid cross-lingual transfer.
  arXiv  Detail & Related papers  (2022-12-20T18:17:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.