Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts

• URL: http://arxiv.org/abs/2604.18473v1
• Date: Mon, 20 Apr 2026 16:24:41 GMT
• Title: Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
• Abstract summary: We present BAR, which trains independent domain experts, each through its own mid-training, supervised finetuning, and reinforcement learning pipeline, and composes them via a Mixture-of-Experts architecture with lightweight router training.<n>At the 7B scale, with experts for math, code, tool use, and safety, BAR achieves an overall score of 49.1.<n>We show that modular training provides a structural advantage: by isolating each domain, it avoids the catastrophic forgetting that occurs when late-stage RL degrades capabilities from earlier training stages, while significantly reducing the cost and complexity of updating or adding a domain.
• Score: 76.12264847243816
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Extending a fully post-trained language model with new domain capabilities is fundamentally limited by monolithic training paradigms: retraining from scratch is expensive and scales poorly, while continued training often degrades existing capabilities. We present BAR (Branch-Adapt-Route), which trains independent domain experts, each through its own mid-training, supervised finetuning, and reinforcement learning pipeline, and composes them via a Mixture-of-Experts architecture with lightweight router training. Unlike retraining approaches that mix all domains and require full reprocessing for any update (with cost scaling quadratically), BAR enables updating individual experts independently with linear cost scaling and no degradation to existing domains. At the 7B scale, with experts for math, code, tool use, and safety, BAR achieves an overall score of 49.1 (averaged across 7 evaluation categories), matching or exceeding re-training baselines (47.8 without mid-training, 50.5 with). We further show that modular training provides a structural advantage: by isolating each domain, it avoids the catastrophic forgetting that occurs when late-stage RL degrades capabilities from earlier training stages, while significantly reducing the cost and complexity of updating or adding a domain. Together, these results suggest that decoupled, expert-based training is a scalable alternative to monolithic retraining for extending language models.

Related papers

• Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model [1.8438975114331797]
  Mixture of Training (MoT) is a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks.<n>MoT trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass.<n>We present MoT as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.
  arXiv  Detail & Related papers  (2026-08-13T14:13:46Z)
• Into the ORBIT for Time Series: Training Regimes for Foundation Models [16.359864017246494]
  ORBIT is a training paradigm that makes pre-training distributions explicit and controllable.<n>We introduce Rank-Guided Cross-Depth Alignment, a training objective that uses late-layer representations as stop-gradient teachers.<n> Evaluations on GIFT-Eval and fev-bench demonstrate strong zero-shot forecasting performance across diverse domains and frequencies.
  arXiv  Detail & Related papers  (2026-08-13T14:00:39Z)
• A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL [49.589717630468876]
  We show that single-domain RL produces sparse, small-magnitude parameter edits with weak overlap among top-changed neurons.<n>We prove under a local model of multi-domain RL that later-domain training harms an earlier domain mainly through a second-order damage term.
  arXiv  Detail & Related papers  (2026-06-01T15:44:56Z)
• Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages [51.86479668480496]
  We tackle the question of how to scale more efficiently across the many, ever-growing stages of current LLM training pipelines.<n>We propose Introspective Training (or IXT), inspired by offline reward-conditioned reinforcement learning.<n>IXT uses a thinking reward model to annotate data with natural language critique based feedback, enabling quality aware training from the earliest stages of the pipeline.
  arXiv  Detail & Related papers  (2026-05-19T06:04:08Z)
• Bagging-Based Model Merging for Robust General Text Embeddings [73.51674133699196]
  General-purpose text embedding models underpin a wide range of NLP and information retrieval applications.<n>We present a systematic study of multi-task training for text embeddings from two perspectives: data scheduling and model merging.<n>We propose Bagging-based rObust mOdel Merging (BOOM), which trains multiple embedding models on sampled subsets and merges them into a single model.
  arXiv  Detail & Related papers  (2026-02-05T15:45:08Z)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models [73.10315509190623]
  Recent reinforcement learning techniques have yielded impressive reasoning improvements in language models.<n>It remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training.<n>We develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training.
  arXiv  Detail & Related papers  (2025-12-08T18:12:10Z)
• EvoLM: In Search of Lost Language Model Training Dynamics [97.69616550374579]
  EvoLM is a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning.<n>By training over 100 LMs with 1B and 4B parameters from scratch, we rigorously evaluate both upstream (language modeling) and downstream (problem-solving) reasoning capabilities.
  arXiv  Detail & Related papers  (2025-06-19T04:58:47Z)
• Optimizing Large Model Training through Overlapped Activation Recomputation [24.28543166026873]
  We present Lynx, a new recomputation framework to reduce overhead by overlapping recomputation with communication in training pipelines.<n>Our comprehensive evaluation using GPT models with 1.3B-23B parameters shows that Lynx outperforms existing recomputation approaches by up to 1.37x.
  arXiv  Detail & Related papers  (2024-06-13T02:31:36Z)
• Boosting Meta-Training with Base Class Information for Few-Shot Learning [35.144099160883606]
  We propose an end-to-end training paradigm consisting of two alternative loops. In the outer loop, we calculate cross entropy loss on the entire training set while updating only the final linear layer. This training paradigm not only converges quickly but also outperforms existing baselines, indicating that information from the overall training set and the meta-learning training paradigm could mutually reinforce one another.
  arXiv  Detail & Related papers  (2024-03-06T05:13:23Z)
• Efficient Stagewise Pretraining via Progressive Subnetworks [53.00045381931778]
  The prevailing view suggests that stagewise dropping strategies, such as layer dropping, are ineffective when compared to stacking-based approaches. This paper challenges this notion by demonstrating that, with proper design, dropping strategies can be competitive, if not better, than stacking methods. We propose an instantiation of this framework - Random Part Training (RAPTR) - that selects and trains only a random subnetwork at each step, progressively increasing the size in stages.
  arXiv  Detail & Related papers  (2024-02-08T18:49:09Z)
• Decoupled Training: Return of Frustratingly Easy Multi-Domain Learning [20.17925272562433]
  Multi-domain learning aims to train a model with minimal average risk across multiple overlapping but non-identical domains. We propose Decoupled Training (D-Train) as a frustratingly easy and hyper parameter-free multi-domain learning method. D-Train is a tri-phase general-to-specific training strategy that first pre-trains on all domains to warm up a root model, then post-trains on each domain by splitting into multi-heads, and finally fine-tunes the heads by fixing the backbone.
  arXiv  Detail & Related papers  (2023-09-19T04:06:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.