SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling

• URL: http://arxiv.org/abs/2606.16456v1
• Date: Mon, 15 Jun 2026 09:28:24 GMT
• Title: SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
• Abstract summary: Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive.<n>MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models.<n>We propose SVD-Partitioned Residual Initialization (SPRI), which distributes SVD-partitioned residuals derived from pretrained feed-forward network weights across routed experts.
• Score: 58.910165627759824
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models. However, existing upcycling methods typically rely on large-scale continued training and often perform poorly under data-constrained supervised adaptation, due to either homogeneous experts or overly disruptive perturbations to pretrained parameters. In this setting, effective upcycling must leverage pretrained weight structure while introducing sufficient diversity among routed experts. To this end, we propose SVD-Partitioned Residual Initialization (SPRI), which distributes SVD-partitioned residuals derived from pretrained feed-forward network (FFN) weights across routed experts, introducing controlled expert diversity grounded in pretrained spectral structure. We further introduce a two-stage training strategy to improve adaptation stability. We evaluate SPRI on multilingual speech-to-text translation, where limited supervised data challenges MoE upcycling and multiple target languages provide natural routing heterogeneity. On CoVoST2 across 15 En-to-XX directions, SPRI improves average BLEU and COMET over fully fine-tuned dense models by 2.58 and 3.32 points, respectively, and outperforms the prior best MoE upcycling baseline by 3.39 BLEU and 4.34 COMET points.

Related papers

• m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encoder [50.2753694138149]
  We introduce m3BERT: a Modern, Multi-lingual, Matryoshka Bidirectional representations.<n>It features a novel pretraining strategy that jointly optimize representations across both transformer layers and multiple embedding dimensions.<n>It significantly outperforms state-of-the-art embedding models in Bing-Click, a large-scale industrial retrieval dataset.
  arXiv  Detail & Related papers  (2026-05-19T09:13:49Z)
• Post-Trained MoE Can Skip Half Experts via Self-Distillation [35.17955778301488]
  Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation.<n>MoE's dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner.<n>This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones.
  arXiv  Detail & Related papers  (2026-05-18T16:50:48Z)
• Mixture of Experts for Low-Resource LLMs [0.0]
  Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood.<n>We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed.
  arXiv  Detail & Related papers  (2026-05-17T18:50:50Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts [27.40662720929157]
  Mixture-of-Experts (MoE) has become the dominant architecture for scaling large language models.<n>We propose expert upcycling, a method for progressively expanding MoE capacity by increasing the number of experts.<n>In our 7B-13B total parameter experiments, the upcycled model matches the fixed-size baseline on validation loss while saving 32% of GPU hours.
  arXiv  Detail & Related papers  (2026-04-21T05:53:33Z)
• Grouter: Decoupling Routing from Representation for Accelerated MoE Training [10.767613437794537]
  Grouter is a preemptive routing method that by distilling high-quality structures from fully-trained MoE models and serving as a fixed router for target models.<n> Experiments demonstrate that Grouter achieves superior performance and efficiency which boosts pre-training data utilization by 4.28x and achieves up to 33.5% throughput acceleration.
  arXiv  Detail & Related papers  (2026-02-22T06:09:57Z)
• Dual Decomposition of Weights and Singular Value Low Rank Adaptation [9.048461365342204]
  We propose DuDe, a novel approach that decomposes weight matrices into magnitude and direction components.<n>Our evaluation demonstrates DuDe's superior performance and robustness, achieving up to 48.35% accuracy on MMLU and 62.53% ($pm$ 1.59) accuracy on GSM8K.
  arXiv  Detail & Related papers  (2025-05-20T13:49:15Z)
• Overtrained Language Models Are Harder to Fine-Tune [64.44743256512237]
  Large language models are pre-trained on ever-growing token budgets.<n>We show that extended pre-training can make models harder to fine-tune, leading to degraded final performance.
  arXiv  Detail & Related papers  (2025-03-24T23:11:56Z)
• Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models [125.91897197446379]
  We find that MoE models benefit more from instruction tuning than dense models. Our most powerful model, FLAN-MOE-32B, surpasses the performance of FLAN-PALM-62B on four benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-24T04:22:26Z)
• Adversarial Self-Attention for Language Understanding [89.265747130584]
  This paper proposes textitAdversarial Self-Attention mechanism (ASA). ASA adversarially reconstructs the Transformer attentions and facilitates model training from contaminated model structures. For fine-tuning, ASA-empowered models consistently outweigh naive models by a large margin considering both generalization and robustness.
  arXiv  Detail & Related papers  (2022-06-25T09:18:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.