Grouter: Decoupling Routing from Representation for Accelerated MoE Training

• URL: http://arxiv.org/abs/2603.06626v1
• Date: Sun, 22 Feb 2026 06:09:57 GMT
• Title: Grouter: Decoupling Routing from Representation for Accelerated MoE Training
• Abstract summary: Grouter is a preemptive routing method that by distilling high-quality structures from fully-trained MoE models and serving as a fixed router for target models.<n> Experiments demonstrate that Grouter achieves superior performance and efficiency which boosts pre-training data utilization by 4.28x and achieves up to 33.5% throughput acceleration.
• Score: 10.767613437794537
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traditional Mixture-of-Experts (MoE) training typically proceeds without any structural priors, effectively requiring the model to simultaneously train expert weights while searching for an optimal routing policy within a vast combinatorial space. This entanglement often leads to sluggish convergence and training instabilities. This paper introduces Grouter, a preemptive routing method that by distilling high-quality structures from fully-trained MoE models and serving as a fixed router for target models. By decoupling structural optimization from weight updates, Grouter significantly accelerates both the speed and quality of model convergence. To ensure the framework's versatility, we also introduce expert folding to adapt Grouter across varying model configurations and expert tuning to rebalance workloads across different data distributions. Furthermore, by leveraging the structural priors provided by preemptive routing, we can implement targeted optimizations to further enhance training throughput. Experiments demonstrate that Grouter achieves superior performance and efficiency which boosts pre-training data utilization by 4.28x and achieves up to 33.5% throughput acceleration, establishing preemptive routing as a fundamental paradigm for scalable MoE training.

Related papers

• Dynamic-in-Few-Step: Unifying Dynamic Computation and Few-Step Distillation for Efficient Video Generation [24.896148135296198]
  Video Diffusion Models (VDMs) have demonstrated superior generation quality but suffer from prohibitive computational costs.<n>We propose a novel post-training acceleration framework that exploits this redundancy by integrating dynamic structural sparsification directly into the distillation process.
  arXiv  Detail & Related papers  (2026-07-07T13:14:33Z)
• SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling [58.910165627759824]
  Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive.<n>MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models.<n>We propose SVD-Partitioned Residual Initialization (SPRI), which distributes SVD-partitioned residuals derived from pretrained feed-forward network weights across routed experts.
  arXiv  Detail & Related papers  (2026-06-15T09:28:24Z)
• SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training [57.41616809842774]
  We systematically study MoE compression in large-scale pretraining.<n>We find that pruning a pretrained MoE consistently outperforms training the target architecture from scratch.<n>We propose multi-token prediction (MTP) distillation, which yields consistent gains.
  arXiv  Detail & Related papers  (2026-05-09T06:50:35Z)
• An Adapter-free Fine-tuning Approach for Tuning 3D Foundation Models [22.63554774942836]
  We propose Momentum-Consistency Fine-Tuning (MCFT) to bridge the gap between full and parameter-efficient fine-tuning.<n>MCFT selectively fine-tunes a portion of the pre-trained encoder while enforcing a momentum-based consistency constraint.<n>We extend MCFT with two variants: a semi-supervised framework that leverages abundant unlabeled data to enhance few-shot performance, and a pruning-based variant that improves computational efficiency.
  arXiv  Detail & Related papers  (2026-03-24T21:28:37Z)
• MeanFlow Transformers with Representation Autoencoders [71.45823902973349]
  MeanFlow (MF) is a diffusion-motivated generative model that enables efficient few-step generation by learning long jumps directly from noise to data.<n>We develop an efficient training and sampling scheme for MF in the latent space of a Representation Autoencoder (RAE)<n>We achieve a 1-step FID of 2.03, outperforming vanilla MF's 3.43, while reducing sampling GFLOPS by 38% and total training cost by 83% on ImageNet 256.
  arXiv  Detail & Related papers  (2025-11-17T06:17:08Z)
• Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts [113.0656076371565]
  We propose a novel router-aware approach to optimize importance sampling weights in off-policy reinforcement learning (RL)<n>Specifically, we design a rescaling strategy guided by router logits, which effectively reduces gradient variance and mitigates training divergence.<n> Experimental results demonstrate that our method significantly improves both the convergence stability and the final performance of MoE models.
  arXiv  Detail & Related papers  (2025-10-27T05:47:48Z)
• Transformer-based Scalable Beamforming Optimization via Deep Residual Learning [12.79709425087431]
  unsupervised deep learning framework for downlink beamforming in large-scale MU-MISO channels.<n>Model is trained offline, allowing real-time inference through lightweight feedforward computations in dynamic communication environments.
  arXiv  Detail & Related papers  (2025-10-15T01:43:51Z)
• MosaicDiff: Training-free Structural Pruning for Diffusion Model Acceleration Reflecting Pretraining Dynamics [34.69318408652807]
  We introduce a novel framework called MosaicDiff that aligns diffusion pretraining dynamics with post-training sampling acceleration.<n>Our method achieves significant speed-ups in sampling without compromising output quality.
  arXiv  Detail & Related papers  (2025-10-13T21:51:04Z)
• Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models [0.41942958779358663]
  We propose a predictive framework that models training dynamics and helps optimize resource usage.<n>We derive an empirical scaling law based on model size, initial performance, and training progress.<n>We find that training beyond certain number of an epoch offers little gain, suggesting earlier stopping can significantly reduce compute without sacrificing performance.
  arXiv  Detail & Related papers  (2025-07-24T01:09:25Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• A Multi-Level Framework for Accelerating Training Transformer Models [5.268960238774481]
  Training large-scale deep learning models poses an unprecedented demand for computing power. We propose a multi-level framework for training acceleration based on Coalescing, De-coalescing and Interpolation. We prove that the proposed framework reduces the computational cost by about 20% on training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model.
  arXiv  Detail & Related papers  (2024-04-07T03:04:34Z)
• Efficient Stagewise Pretraining via Progressive Subnetworks [53.00045381931778]
  The prevailing view suggests that stagewise dropping strategies, such as layer dropping, are ineffective when compared to stacking-based approaches. This paper challenges this notion by demonstrating that, with proper design, dropping strategies can be competitive, if not better, than stacking methods. We propose an instantiation of this framework - Random Part Training (RAPTR) - that selects and trains only a random subnetwork at each step, progressively increasing the size in stages.
  arXiv  Detail & Related papers  (2024-02-08T18:49:09Z)
• Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models [125.91897197446379]
  We find that MoE models benefit more from instruction tuning than dense models. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-24T04:22:26Z)
• Adversarial Self-Attention for Language Understanding [89.265747130584]
  This paper proposes textitAdversarial Self-Attention mechanism (ASA). ASA adversarially reconstructs the Transformer attentions and facilitates model training from contaminated model structures. For fine-tuning, ASA-empowered models consistently outweigh naive models by a large margin considering both generalization and robustness.
  arXiv  Detail & Related papers  (2022-06-25T09:18:10Z)
• Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free [115.81899803240758]
  Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. This paper asks how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies. Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework.
  arXiv  Detail & Related papers  (2020-10-22T16:06:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.