Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration

• URL: http://arxiv.org/abs/2608.28511v1
• Date: Fri, 28 Aug 2026 16:44:50 GMT
• Title: Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
• Abstract summary: Communication-efficient MoE models (CE-MoE)<n>Adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth.<n>At the 31.5B scale, CE-MoE uses 33.3% fewer GPU-hours while improving average downstream score and inference throughput.
• Score: 13.010351659575297
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters, CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks with full-MoE baselines. At the 31.5B scale, CE-MoE uses 33.3\% fewer GPU-hours while improving average downstream score and inference throughput.

Related papers

• MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models [55.732137666068446]
  Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities.<n>MoEMB scales UME along the expert axis through mixture-of-experts (MoE)<n>MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes.
  arXiv  Detail & Related papers  (2026-09-08T12:29:50Z)
• Instella-MoE Technical Report [22.767094992305257]
  We introduce Instella-MoE, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token.<n>Instella-MoE is trained entirely from scratch on AMD Instinct MI300X and MI325X graphics cards.
  arXiv  Detail & Related papers  (2026-09-01T06:38:17Z)
• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models [129.64558694024126]
  Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling.<n>We characterize how optimization hyper parameters, compute allocation, and architecture scale for MoE dLLMs.<n>We train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens.
  arXiv  Detail & Related papers  (2026-08-04T10:53:02Z)
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping [52.02659589971978]
  We propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference.<n>MoDES significantly enhances inference speed, improving the prefilling time by 2.16$times$ and the decoding time by 1.26$times$.
  arXiv  Detail & Related papers  (2025-11-19T18:48:27Z)
• HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap [17.1806530983927]
  We introduce HierMoE to accelerate the training of large language models (LLMs) by two topology-aware techniques.<n>Our prototype HierMoE achieves $1.55times$ to $3.32times$ faster communication and delivers $1.18times$ to $1.27times$ faster end-to-end training.
  arXiv  Detail & Related papers  (2025-08-13T08:16:31Z)
• FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models [21.96960353910023]
  We introduce FSMoE, a flexible training system optimizing task scheduling with three novel techniques.<n>We conduct extensive experiments with configured MoE layers and real-world MoE models on two GPU clusters.<n> FSMoE supports four popular types of MoE routing functions and is more efficient than existing implementations.
  arXiv  Detail & Related papers  (2025-01-18T10:14:37Z)
• Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models [62.4691912312317]
  Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4$times$ compared to dense models without sacrificing performance. We propose a hybrid dense training and sparse inference framework for MoE models (DS-MoE) which achieves strong computation and parameter efficiency.
  arXiv  Detail & Related papers  (2024-04-08T14:39:49Z)
• LocMoE: A Low-Overhead MoE for Large Language Model Training [13.153904674287546]
  We propose a novel routing strategy that combines load balance and locality by converting partial inter-node communication to that of intra-node. The proposed LocMoE reduces training time per epoch by 12.68% to 22.24% compared to classical routers.
  arXiv  Detail & Related papers  (2024-01-25T03:36:39Z)
• Mixture of Tokens: Continuous MoE through Cross-Example Aggregation [0.7880651741080428]
  Mixture of Experts (MoE) models are pushing the boundaries of language and vision tasks. MoT is a simple, continuous architecture that is capable of scaling the number of parameters similarly to sparse MoE models. Our best models achieve a 3x increase in training speed over dense Transformer models in language pretraining.
  arXiv  Detail & Related papers  (2023-10-24T16:03:57Z)
• Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism [91.9372563527801]
  Existing MoE models suffer from tremendous inner-node and inter-node communication overhead. We propose a novel MoE architecture called Pipeline MoE (PPMoE) to tackle them. PPMoE builds expert parallel incorporating with tensor parallel and replaces communication-intensive all-to-all dispatching and gathering.
  arXiv  Detail & Related papers  (2023-04-22T14:09:14Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)
• Sparse MoEs meet Efficient Ensembles [49.313497379189315]
  We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs) We present Efficient Ensemble of Experts (E$3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble.
  arXiv  Detail & Related papers  (2021-10-07T11:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.