Temporally Extended Mixture-of-Experts Models

• URL: http://arxiv.org/abs/2604.20156v1
• Date: Wed, 22 Apr 2026 03:50:52 GMT
• Title: Temporally Extended Mixture-of-Experts Models
• Abstract summary: Mixture-of-Experts models are popular for scaling capacity at fixed inference speed.<n>Once a model outgrows available GPU memory, this churn can render optimizations like offloading and pre-fetching ineffective.<n>We make the case that the options framework in reinforcement learning is a perfect match to tackle this problem, and argue for temporally extended mixture-of-experts layers.
• Score: 6.07059176899521
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts models, now popular for scaling capacity at fixed inference speed, switch experts at nearly every token. Once a model outgrows available GPU memory, this churn can render optimizations like offloading and pre-fetching ineffective. We make the case that the options framework in reinforcement learning is a perfect match to tackle this problem, and argue for temporally extended mixture-of-experts layers. Building on the option-critic framework with deliberation costs, we add a controller to each layer that learns when to switch expert sets and which to load. By applying this to gpt-oss-20b with low-rank adapters and a self-distillation reward, our method reduces switch rates from over 50% to below 5% while retaining up to 90% of base-model accuracy on MATH, MMLU, and MMMLU. This shows that even existing pre-trained models can be converted to temporally extended MoEs with lightweight training, with the deliberation cost allowing model trainers to trade off switching rates against capability. We hope this opens a principled path, grounded in the options framework, for memory-efficient serving and continual learning in ever-growing MoE models.

Related papers

• Dynamic Mixture of Latent Memories for Self-Evolving Agents [57.20419419302731]
  MoLEM is a generative mixture of latent memory framework based on a dynamic mixture-of-experts (MoE)<n>We train the framework on continual-learning sequences spanning math, science, and code domains.<n>After the full continual-learning sequence, our method improves the average accuracy by 10.40% over the Vanilla pretrained baseline.
  arXiv  Detail & Related papers  (2026-05-21T03:35:10Z)
• Scalable Training of Mixture-of-Experts Models with Megatron Core [26.9162079065285]
  Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models.<n>Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation.<n>We address these challenges for MoE training through integrated optimizations spanning memory (fine-dimensional recomputation, offloading), communication, and computation.
  arXiv  Detail & Related papers  (2026-03-08T15:42:43Z)
• MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models [6.372179935695467]
  Training large-scale Mixture-of-Experts (MoE) models requires high-memory, high-bandwidth GPUs (e.g., A100)<n>MoE-DisCo decomposes the MoE model into multiple dense submodels, each consisting of a shared backbone and a single expert, and partitions the training data into subsets using unsupervised clustering.
  arXiv  Detail & Related papers  (2026-01-11T10:59:15Z)
• Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization [60.309915093470416]
  Matryoshka MoE (M-MoE) is a training framework that instills a coarse-to-fine structure directly into the expert ensemble.<n>Our work paves the way for more practical and adaptable deployments of large-scale MoE models.
  arXiv  Detail & Related papers  (2025-09-30T16:56:44Z)
• SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling [1.2777855412373709]
  Mixture-of-Experts (MoE) models have become a widely-adopted solution to continue scaling model sizes without a corresponding linear increase in compute.<n>Current systems are forced to either drop tokens assigned to popular experts, degrading convergence, or frequently rebalance resources allocated to each expert based on popularity.<n>We introduce SYMI, an adaptive MoE training system.
  arXiv  Detail & Related papers  (2025-04-28T15:58:55Z)
• Towards Optimal Heterogeneous Client Sampling in Multi-Model Federated Learning [22.787635207005884]
  Federated learning allows edge devices to collaboratively train models without sharing local data.<n>Clients may need to train multiple unrelated FL models, but communication constraints limit their ability to train all models simultaneously.<n>We propose MMFL-LVR, a loss-based sampling method that minimizes training variance while explicitly respecting communication limits at the server.
  arXiv  Detail & Related papers  (2025-04-07T14:43:17Z)
• Muon is Scalable for LLM Training [50.68746986439438]
  We introduce Moonlight, a Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon.<n>Our model improves the current frontier, achieving better performance with much fewer training FLOPs compared to prior models.<n>We open-source our distributed Muon implementation that is memory optimal and communication efficient.
  arXiv  Detail & Related papers  (2025-02-24T09:12:29Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models [31.960749305728488]
  We introduce a novel concept dubbed modular neural tangent kernel (mNTK) We show that the quality of a module's learning is tightly associated with its mNTK's principal eigenvalue $lambda_max$. We propose a novel training strategy termed Modular Adaptive Training (MAT) to update those modules with their $lambda_max$ exceeding a dynamic threshold.
  arXiv  Detail & Related papers  (2024-05-13T07:46:48Z)
• Toward Inference-optimal Mixture-of-Expert Large Language Models [55.96674056805708]
  We study the scaling law of MoE-based large language models (LLMs) We find that MoEs with a few (4/8) experts are the most serving efficient solution under the same performance, but costs 2.5-3.5x more in training. We propose to amend the scaling law of MoE by introducing inference efficiency as another metric besides the validation loss.
  arXiv  Detail & Related papers  (2024-04-03T16:33:42Z)
• Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters [65.15700861265432]
  We present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters. To preserve the zero-shot recognition capability of vision-language models, we introduce a Distribution Discriminative Auto-Selector.
  arXiv  Detail & Related papers  (2024-03-18T08:00:23Z)
• LocMoE: A Low-Overhead MoE for Large Language Model Training [13.153904674287546]
  We propose a novel routing strategy that combines load balance and locality by converting partial inter-node communication to that of intra-node. The proposed LocMoE reduces training time per epoch by 12.68% to 22.24% compared to classical routers.
  arXiv  Detail & Related papers  (2024-01-25T03:36:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.