Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models

• URL: http://arxiv.org/abs/2608.07890v1
• Date: Sat, 08 Aug 2026 03:36:41 GMT
• Title: Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
• Abstract summary: Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert.<n>Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning.<n>We test whether lightweight adaptation can recover this signal.
• Score: 3.747647072414422
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert. Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning. We test whether lightweight adaptation can recover this signal. We briefly fine-tune with a parameter-efficient adapter, rank experts by the induced $\ell_2$ router change, and prune the least-changed experts in one shot. On Mixtral-8$\times$7B-Instruct (44.83% MMLU-Pro), router-only LoRA trains 0.002% of parameters and outperforms all-module LoRA at matched rank with half the experts removed (27.54% vs. 24.42%); signal quality declines as adaptation spreads to attention and expert weights. Accuracy improves monotonically with LoRA rank, reaching 28.76%. IA3, which leaves router weights frozen, matches direct router adaptation, whereas unconstrained additive adapters degrade the signal. Router-guided MMLU-Pro accuracy decays quasi-linearly rather than collapsing, remains nearly 1.8 times that of magnitude-based or random pruning at maximal compression, and reduces memory by 49% and per-token latency by 37%. At 25% compression, retention is competitive with methods using full activation statistics. The criterion also transfers to Qwen1.5-MoE fine-tuned for mathematics, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed while random pruning falls to single digits. Router sensitivity under lightweight fine-tuning therefore makes provably motivated expert pruning practical at scale.

Related papers

• ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs [15.24786135265898]
  Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs)<n>We propose ACE, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping.
  arXiv  Detail & Related papers  (2026-09-04T14:56:00Z)
• When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models [14.959514582161956]
  Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models.<n>We observe that this assumption breaks down under over-dispersed routing, in which tokens are distributed nearly uniformly across experts.<n>We propose Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving whichever domain is currently worst-affected.
  arXiv  Detail & Related papers  (2026-09-03T20:21:20Z)
• Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs [8.622261628194893]
  Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token sparse.<n>We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias.
  arXiv  Detail & Related papers  (2026-08-27T06:01:46Z)
• BoostLoRA: Growing Effective Rank by Boosting Adapters [3.022272657109967]
  BoostLoRA is a gradient-boosting framework that overcomes the limit by iteratively training and merging minimal adapters.<n>It is the first PEFT method whose effective rank grows with training, separating per-round parameter cost from total representational capacity.
  arXiv  Detail & Related papers  (2026-04-30T01:43:00Z)
• MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning [0.5076419064097734]
  MoE-Sieve is a simple routing-guided framework for LoRA fine-tuning.<n>We show that tuning only the top 25% routed experts per layer remains competitive with full LoRA.<n>We also observe a non-monotonic relationship between expert count and seed-to-seed variance.
  arXiv  Detail & Related papers  (2026-03-25T07:53:09Z)
• Directional Routing in Transformers [0.0]
  directional routing is a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router.<n>We trace the resulting circuits through mechanistic interpretability.<n> routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.
  arXiv  Detail & Related papers  (2026-03-16T07:28:22Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts [74.40169987564724]
  Expert parallelism (EP) is designed to scale MoE models by distributing experts across multiple devices.<n>Under extreme imbalance, EP can funnel a disproportionate number of tokens to a small number of experts, leading to compute- and memory-bound failures.<n>We propose Least-Loaded Expert Parallelism (LLEP), a novel EP algorithm that dynamically reroutes excess tokens and associated expert parameters from overloaded devices to underutilized ones.
  arXiv  Detail & Related papers  (2026-01-23T18:19:15Z)
• Uncertainty Makes It Stable: Curiosity-Driven Quantized Mixture-of-Experts [6.221156050218661]
  We present a curiosity-driven quantized Mixture-of-Experts framework for deep neural networks on resource-constrained devices.<n>Our 4-bit quantization maintains 99.9 percent of 16-bit accuracy (0.858 vs 0.859 F1) with 4x compression and 41 percent energy savings.<n>Our information-theoretic routing demonstrates that adaptive quantization yields accurate (0.858 F1, 1.2M params), energy-efficient (3.87 F1/mJ), and predictable edge models.
  arXiv  Detail & Related papers  (2025-11-13T15:32:41Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• MC#: Mixture Compressor for Mixture-of-Experts Large Models [86.64315380917827]
  Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation.<n>We propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning.
  arXiv  Detail & Related papers  (2025-10-13T03:12:46Z)
• From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing [52.01745035243826]
  Mixture-of-Experts (MoE) models can scale parameter capacity by routing each token to a subset of experts.<n> conditional routing shifts the burden on inference memory, limiting the number of experts per device.<n>We present LASER, a plug-and-play, inference-time routing algorithm that balances load while preserving accuracy.
  arXiv  Detail & Related papers  (2025-09-29T16:29:17Z)
• Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models [1.7272658301768147]
  MoE-MLA-RoPE is a novel architecture combination that combines Mixture of Experts (MoE) with Multi-head Latent Attention (MLA) and Rotary Position Embeddings (RoPE) for efficient language modeling.<n>Our approach addresses the fundamental trade-off between model capacity and computational efficiency through three key innovations.
  arXiv  Detail & Related papers  (2025-08-02T08:33:30Z)
• Retrosynthetic Planning with Dual Value Networks [107.97218669277913]
  We propose a novel online training algorithm, called Planning with Dual Value Networks (PDVN) PDVN alternates between the planning phase and updating phase to predict the synthesizability and cost of molecules. On the widely-used USPTO dataset, our PDVN algorithm improves the search success rate of existing multi-step planners.
  arXiv  Detail & Related papers  (2023-01-31T16:43:53Z)
• Non-Parametric Adaptive Network Pruning [125.4414216272874]
  We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
  arXiv  Detail & Related papers  (2021-01-20T06:18:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.