SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs

• URL: http://arxiv.org/abs/2606.17952v1
• Date: Tue, 16 Jun 2026 14:05:41 GMT
• Title: SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
• Abstract summary: We propose SoftMoE, which replaces discrete routing with a truncated soft top-$k$ LapSum relaxation.<n>We further parameterize the mean number of active experts per layer and impose a global budget constraint.<n>SoftMoE remains fully compatible with autoregressive modeling and performance comparable to or better than sparse MoE on language modeling and downstream tasks.
• Score: 10.607329413550223
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-$k$ routing. While this preserves causality and suits autoregressive language models, the discrete top-$k$ operator is not differentiable, forcing a fixed number of active experts per input and resulting in inefficient use of computation. We propose SoftMoE, which replaces discrete routing with a truncated soft top-$k$ LapSum relaxation, allowing gradient-based optimization of expert routing. We further parameterize the mean number of active experts per layer and impose a global budget constraint, enabling the model to learn how to allocate expert capacity across layers. SoftMoE remains fully compatible with autoregressive modeling and achieves performance comparable to or better than sparse MoE on language modeling and downstream tasks, while activating significantly fewer experts. Notably, the learned allocation is highly non-uniform, with later layers activating more experts. The source code is publicly available$^\dagger$.

Related papers

• REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models [15.630466998208261]
  Mixture-of-experts (MoE) models increase parameter capacity by activating only a small subset of experts for each token.<n>In diffusion language models (DLMs), however, each denoising forward jointly revisits all token positions despite their sharply different refinement demands.<n>We argue that MoE inference in DLMs should therefore be viewed as refinement-aware compute allocation across heterogeneous token refinement states.
  arXiv  Detail & Related papers  (2026-08-03T06:59:06Z)
• Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models [50.34585122347149]
  We introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers.<n>We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs.
  arXiv  Detail & Related papers  (2026-06-15T15:08:09Z)
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping [52.02659589971978]
  We propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference.<n>MoDES significantly enhances inference speed, improving the prefilling time by 2.16$times$ and the decoding time by 1.26$times$.
  arXiv  Detail & Related papers  (2025-11-19T18:48:27Z)
• ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts [25.46805026086543]
  We describe ReXMoE, a novel MoE architecture that improves routing beyond the existing layer-local approaches.<n>ReXMoE decouples expert dimensionality from per-layer budgets, enabling richer expert combinations without sacrificing individual expert capacity.
  arXiv  Detail & Related papers  (2025-10-20T12:27:55Z)
• L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts [10.21556794551883]
  We present L-MoE: a Lightweight Mixture of LoRA Experts.<n>L-MoE redefines MoE experts as task-specialized, low-rank adapters.<n>We present the formal mathematical framework for L-MoE.
  arXiv  Detail & Related papers  (2025-10-19T08:44:25Z)
• Cluster Topology-Driven Placement of Experts Reduces Network Traffic in MoE Inference [49.141930185079325]
  We propose an integer linear program (ILP) that determines the optimal placement of experts, minimizing the expected number of transmissions.<n>We demonstrate that ILP-based placement strategy yields lower network traffic than competitors for small-scale (DeepSeekMoE16B) and large-scale (DeepSeek-R1671B) models.
  arXiv  Detail & Related papers  (2025-08-12T07:08:48Z)
• DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs [86.76714527437383]
  This paper proposes DSMoE, a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks.<n>We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge.<n>Experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches.
  arXiv  Detail & Related papers  (2025-02-18T02:37:26Z)
• GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration [39.302800055216764]
  We introduce a novel MoE graph-based framework $textbfGMoE$, aimed at enhancing the collaboration among multiple experts.<n>In GMoE, a graph router function is designed to capture the collaboration signals among experts.<n>We put forward two coordination strategies in GMoE: the $textitPoisson distribution-based distinction strategy and the $textitNormal distribution-based balance strategy.
  arXiv  Detail & Related papers  (2024-12-18T02:18:57Z)
• MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition [32.97035551579975]
  We introduce a two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost. Experiments on Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Mixtral-8$times$7B demonstrate that our proposed methods can both reduce the model size and enhance inference efficiency.
  arXiv  Detail & Related papers  (2024-11-01T20:37:58Z)
• Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models [90.14693869269519]
  MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. This paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques.
  arXiv  Detail & Related papers  (2024-02-22T18:56:07Z)
• Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization [51.98792406392873]
  Mixture of Experts (MoE) provides a powerful way to decompose dense layers into smaller, modular computations. A major challenge lies in the computational cost of scaling the number of experts high enough to achieve fine-grained specialization. We propose the Multilinear Mixture of Experts ($mu$MoE) layer to address this, focusing on vision models.
  arXiv  Detail & Related papers  (2024-02-19T21:20:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.