Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

• URL: http://arxiv.org/abs/2608.26650v1
• Date: Thu, 27 Aug 2026 06:01:46 GMT
• Title: Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs
• Abstract summary: Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token sparse.<n>We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias.
• Score: 8.622261628194893
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. In many deployed MoEs, the number of active experts is fixed across layers and tasks, although layer roles and expert redundancy vary with depth and demand varies with difficulty. Existing approaches address only part of this setting: layer-wise allocations are usually determined offline and reused for all tasks, while token-level methods vary expert activation using local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias. The backbone, experts, and router remain frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy-expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) and achieves comparable MMLU accuracy (0.489 vs. 0.474), whereas an aggressive setting activates 2.28 experts on average (62% fewer) with accuracy approximately 3.7 percentage points lower. The MMLU-trained controller also transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.

Related papers

• ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs [15.24786135265898]
  Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs)<n>We propose ACE, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping.
  arXiv  Detail & Related papers  (2026-09-04T14:56:00Z)
• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models [3.747647072414422]
  Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert.<n>Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning.<n>We test whether lightweight adaptation can recover this signal.
  arXiv  Detail & Related papers  (2026-08-08T03:36:41Z)
• AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding [21.76650680783374]
  Speculative decoding verifies a tree of draft tokens in one target-model forward pass.<n>For a mixture-of-experts (MoE) target, parallel verification can activate the union of the experts selected by all tree nodes.<n>We introduce AcceptMoE, a verifier-side expert selector that automatically adjusts the number of eligible experts for each verification block.
  arXiv  Detail & Related papers  (2026-08-04T01:01:12Z)
• Sticky Routing: Training MoE Models for Memory-Efficient Inference [0.0]
  StickyMoE penalises abrupt expert switches between adjacent tokens.<n>Unlike post-hoc methods, StickyMoE allows expert representations and routing decisions to co-adapt from the first training step.
  arXiv  Detail & Related papers  (2026-06-12T23:47:19Z)
• Hierarchical Mixture-of-Experts with Two-Stage Optimization [84.70724165894501]
  We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels.<n>We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and vision benchmarks.<n>In large-scale pre-training on 58B tokens, Hi-MoE-7B achieves a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B.
  arXiv  Detail & Related papers  (2026-05-08T09:21:46Z)
• Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns [10.028141800987548]
  MoE inference at scale is bottlenecked by expert load imbalance and inefficient token routing.<n>We study SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B.<n>Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy.
  arXiv  Detail & Related papers  (2026-04-25T05:33:03Z)
• MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning [0.5076419064097734]
  MoE-Sieve is a simple routing-guided framework for LoRA fine-tuning.<n>We show that tuning only the top 25% routed experts per layer remains competitive with full LoRA.<n>We also observe a non-monotonic relationship between expert count and seed-to-seed variance.
  arXiv  Detail & Related papers  (2026-03-25T07:53:09Z)
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning [83.66308307152808]
  We propose StAbilized Mixture-of-Experts (SAME) for Multimodal Continual Instruction Tuning (MCIT)<n>SAME stabilizes expert selection by decomposing routing dynamics into subspaces and updating only task-relevant directions.<n>It also introduces adaptive expert activation to freeze selected experts during training, reducing redundant and cross-task interference.
  arXiv  Detail & Related papers  (2026-02-02T11:47:06Z)
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping [52.02659589971978]
  We propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference.<n>MoDES significantly enhances inference speed, improving the prefilling time by 2.16$times$ and the decoding time by 1.26$times$.
  arXiv  Detail & Related papers  (2025-11-19T18:48:27Z)
• Mixtral of Experts [57.411379935325435]
  Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model. Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks.
  arXiv  Detail & Related papers  (2024-01-08T18:47:34Z)
• MoEC: Mixture of Expert Clusters [93.63738535295866]
  Sparsely Mixture of Experts (MoE) has received great interest due to its promising scaling capability with affordable computational overhead. MoE converts dense layers into sparse experts, and utilizes a gated routing network to make experts conditionally activated. However, as the number of experts grows, MoE with outrageous parameters suffers from overfitting and sparse data allocation.
  arXiv  Detail & Related papers  (2022-07-19T06:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.