MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning

• URL: http://arxiv.org/abs/2603.24044v1
• Date: Wed, 25 Mar 2026 07:53:09 GMT
• Title: MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning
• Abstract summary: MoE-Sieve is a simple routing-guided framework for LoRA fine-tuning.<n>We show that tuning only the top 25% routed experts per layer remains competitive with full LoRA.<n>We also observe a non-monotonic relationship between expert count and seed-to-seed variance.
• Score: 0.5076419064097734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard LoRA fine-tuning of Mixture-of-Experts (MoE) models applies adapters to every expert, yet our profiling shows that per-layer expert routing is highly skewed: a small subset of experts handles most tokens in each layer, while many others are rarely activated ("cold"). We propose MoE-Sieve, a simple routing-guided framework for LoRA fine-tuning, and pair it with a systematic profiling study of expert routing across architectures and tasks. The method is simple: profile routing counts on a small calibration set, select the top-k most-routed experts per layer, and apply LoRA only to those experts. Across two architecturally distinct MoE models and three diverse tasks, tuning only the top 25% routed experts per layer remains competitive with full LoRA, with mean differences within +/-1 percentage point across all conditions. This reduces LoRA trainable parameters by 70-73%, adapter checkpoint size by 71-73%, and wall-clock training time by up to 50%. We also observe a non-monotonic relationship between expert count and seed-to-seed variance, consistent with the hypothesis that adapting cold experts can introduce gradient noise without improving accuracy. Further ablations show that random expert selection at matched budget is about 2.5 percentage points worse, indicating that the routing signal matters, while greedy per-layer budget optimization does not improve over uniform top-k.

Related papers

• ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs [15.24786135265898]
  Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs)<n>We propose ACE, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping.
  arXiv  Detail & Related papers  (2026-09-04T14:56:00Z)
• Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs [8.622261628194893]
  Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token sparse.<n>We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias.
  arXiv  Detail & Related papers  (2026-08-27T06:01:46Z)
• Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection [0.0]
  Mixture-of-Experts (MoE) models enable model scaling while maintaining low inference-time compute.<n>We introduce elbow-based routing, a training-free inference-time modification that dynamically adjusts the number of experts on a per-token basis.<n> Experiments on a state-of-the-art MoE model demonstrate an average latency reduction of 5.3% while maintaining accuracy across six benchmarks.
  arXiv  Detail & Related papers  (2026-08-05T03:10:17Z)
• Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs [27.71932231964013]
  We propose a framework for composing frozen reasoning LoRA experts through unit-scale hard selection.<n>Experiments show that Hard-Routed MoR-LoRA preserves expert behavior while requiring substantially fewer trainable parameters than soft-routing mixture baselines.
  arXiv  Detail & Related papers  (2026-06-30T09:40:38Z)
• Slicing and Dicing: Configuring Optimal Mixtures of Experts [49.5613403644084]
  We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters.<n>We exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms.<n>Our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.
  arXiv  Detail & Related papers  (2026-05-12T07:47:22Z)
• Hierarchical Mixture-of-Experts with Two-Stage Optimization [84.70724165894501]
  We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels.<n>We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and vision benchmarks.<n>In large-scale pre-training on 58B tokens, Hi-MoE-7B achieves a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B.
  arXiv  Detail & Related papers  (2026-05-08T09:21:46Z)
• CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging [49.87105462292961]
  Core Space Mixture of LoRA (bfCoMoL) is a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation.<n>CoMoL consistently outperforms existing methods across multiple tasks.
  arXiv  Detail & Related papers  (2026-02-28T09:40:11Z)
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping [52.02659589971978]
  We propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference.<n>MoDES significantly enhances inference speed, improving the prefilling time by 2.16$times$ and the decoding time by 1.26$times$.
  arXiv  Detail & Related papers  (2025-11-19T18:48:27Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• Hierarchical LoRA MoE for Efficient CTR Model Scaling [56.608809143548946]
  HiLoMoE is a hierarchical LoRA MoE framework that enables holistic scaling in a parameter-efficient manner.<n>Unlike conventional stacking, HiLoMoE routes based on prior layer scores rather than outputs, allowing all layers to execute in parallel.
  arXiv  Detail & Related papers  (2025-10-12T03:54:11Z)
• LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts [24.0422448103907]
  We propose LD-MoLE, a Learnable Dynamic routing mechanism for Mixture of LoRA Experts.<n>Our design allows the model to adaptively determine the number of experts to activate for each token at different layers.<n>Our method achieves superior performance, but also demonstrates the ability to learn token-dependent and layer-wise expert allocation.
  arXiv  Detail & Related papers  (2025-09-30T02:38:10Z)
• GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors [29.157019630756654]
  Low-Rank Adaptation (LoRA) is an efficient way to adapt large language models with reduced computational costs.<n>GuiLoMo is a fine-grained layer-wise expert numbers and ranks allocation strategy.
  arXiv  Detail & Related papers  (2025-06-17T15:41:33Z)
• LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing [17.171872354057694]
  We propose the LoRA-Mixer, a modular and lightweight MoE framework that integrates LoRA experts.<n>Our core innovation lies in replacing the projection matrices of the attention module's input/output linear layers with task-specific LoRA experts.<n>LoRA-Mixer achieves significant improvements on datasets such as GSM8K, HumanEval, and MedQA.
  arXiv  Detail & Related papers  (2025-06-17T14:58:54Z)
• AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality [31.830108790753172]
  Low-Rank Adaptation (LoRA) is known to enhance training efficiency in Large Language Models (LLMs) Recent studies seek to combine LoRA with Mixture-of-Experts (MoE) to boost performance across various tasks. We introduce AlphaLoRA, a theoretically principled and training-free method for allocating LoRA experts to further redundancy.
  arXiv  Detail & Related papers  (2024-10-14T00:43:02Z)
• Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training [73.90260246781435]
  We present Lory, the first approach that scales such architectures to autoregressive language model pre-training. We show significant performance gains over parameter-matched dense models on both perplexity and a variety of downstream tasks. Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing.
  arXiv  Detail & Related papers  (2024-05-06T03:06:33Z)
• SiRA: Sparse Mixture of Low Rank Adaptation [63.926732717719354]
  We investigate the importance of leveraging "sparse" computation and propose SiRA: sparse mixture of low rank. Specifically it enforces the top $k$ experts routing with a capacity limit restricting the maximum number of tokens each expert can process.
  arXiv  Detail & Related papers  (2023-11-15T18:15:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.