Mixture of Heterogeneous Grouped Experts for Language Modeling

• URL: http://arxiv.org/abs/2604.23108v2
• Date: Tue, 28 Apr 2026 02:47:36 GMT
• Title: Mixture of Heterogeneous Grouped Experts for Language Modeling
• Abstract summary: Large Language Models (LLMs) based on Mixture-of-Experts (MoE) are pivotal in industrial applications for their ability to scale performance efficiently.<n>Standard MoEs enforce uniform expert sizes, creating a rigidity that fails to align computational costs with varying token-level complexity.<n>We propose Mixture of Heterogeneous Grouped Experts (MoHGE) which introduces a two-level routing mechanism to enable flexible, resource-aware expert combinations.
• Score: 19.29654468661715
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language Models (LLMs) based on Mixture-of-Experts (MoE) are pivotal in industrial applications for their ability to scale performance efficiently. However, standard MoEs enforce uniform expert sizes,creating a rigidity that fails to align computational costs with varying token-level complexity. While heterogeneous expert architectures attempt to address this by diversifying expert sizes, they often suffer from significant system-level challenges, specifically unbalanced GPU utilization and inefficient parameter utilization, which hinder practical deployment. To bridge the gap between theoretical heterogeneity and robust industrial application, we propose Mixture of Heterogeneous Grouped Experts (MoHGE) which introduces a two-level routing mechanism to enable flexible, resource-aware expert combinations. To optimize inference efficiency, we propose a Group-Wise Auxiliary Loss, which dynamically steers tokens to the most parameter-efficient expert groups based on task difficulty. To address the critical deployment challenge of GPU load balancing, we introduce an All-size Group-decoupling Allocation strategy coupled with an Intra-Group Experts Auxiliary Loss. These mechanisms collectively ensure uniform computation distribution across GPUs. Extensive evaluations demonstrate that MoHGE matches the performance of MoE architectures while reducing the total parameters by approximately 20% and maintaining balanced GPU utilization. Our work establishes a scalable paradigm for resource-efficient MoE design, offering a practical solution for optimizing inference costs in real-world scenarios. The code is publicly available at https://github.com/UnicomAI/MoHGE.

Related papers

• MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models [55.732137666068446]
  Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities.<n>MoEMB scales UME along the expert axis through mixture-of-experts (MoE)<n>MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes.
  arXiv  Detail & Related papers  (2026-09-08T12:29:50Z)
• Hybrid Workflow Composition for Extreme-Scale Data Processing: A Case Study on the HL-LHC (Extended Version) [0.5156484100374059]
  This paper presents a novel simulation framework for characterizing the interplay between taskset granularity and system-level constraints.<n>Results show that hybrid composition strategies can yield up to 3.8x throughput increase and a 14.9x reduction in network overhead.
  arXiv  Detail & Related papers  (2026-07-29T13:07:31Z)
• E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments [2.0144904127859236]
  Large Language Models (LLMs) have become integral to modern applications, yet their deployment remains challenging.<n>We introduce E2LLM, a framework designed to enable efficient LLM deployment in resource limited settings.
  arXiv  Detail & Related papers  (2026-06-02T15:23:28Z)
• DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts [56.175408382091796]
  We show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router.<n>We propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts.
  arXiv  Detail & Related papers  (2026-05-31T07:08:16Z)
• Token-Level LLM Collaboration via FusionRoute [60.72307345997823]
  FusionRoute is a token-level multi-LLM collaboration framework.<n>It selects the most suitable expert at each decoding step and contributes a complementary logit that refines or corrects the selected expert's next-token distribution.<n>It outperforms both sequence- and token-level collaboration, model merging, and direct fine-tuning.
  arXiv  Detail & Related papers  (2026-01-08T16:53:16Z)
• HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts [26.55877320740609]
  We propose HFedMoE, a heterogeneous MoE-based FL fine-tuning framework that customizes a subset of experts to each client.<n> HFedMoE identifies the expert importance based on its contributions to fine-tuning performance.<n>It then adaptively selects a subset of experts from an information bottleneck perspective to align with each client's computing budget.
  arXiv  Detail & Related papers  (2026-01-02T05:56:11Z)
• Mixture of Ranks with Degradation-Aware Routing for One-Step Real-World Image Super-Resolution [76.66229730098759]
  In real-world image super-resolution (Real-ISR), existing approaches mainly rely on fine-tuning pre-trained diffusion models.<n>We propose a Mixture-of-Ranks (MoR) architecture for single-step image super-resolution.<n>We introduce a fine-grained expert partitioning strategy that treats each rank in LoRA as an independent expert.
  arXiv  Detail & Related papers  (2025-11-20T04:11:44Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference [5.786961198115219]
  Serving Large Language Models (LLMs) is a GPU-intensive task where traditional autoscalers fall short.<n>We introduce HeteroScale, a coordinated autoscaling framework that addresses the core challenges of P/D disaggregated serving.
  arXiv  Detail & Related papers  (2025-08-27T04:22:02Z)
• FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoE [21.860699562235776]
  FLAME is a novel federated learning framework based on the Sparse Mixture-of-Experts (SMoE) architecture.<n>It retains full (uncompressed) global LoRA matrices and achieves client-side adaptability by varying the number of activated experts per client.<n>It tackles these challenges through a lightweight rescaling mechanism and an activation-aware aggregation scheme.
  arXiv  Detail & Related papers  (2025-06-19T21:02:19Z)
• MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models [61.89384981175277]
  We propose a emphheterogeneous textbfMixture-of-Adapters (MoA) approach to integrate Low-Rank Adaptation (LoRA) and Mixture-of-Experts (MoE)<n> Experimental results demonstrate that heterogeneous MoA outperforms homogeneous MoE-LoRA methods in both performance and parameter efficiency.
  arXiv  Detail & Related papers  (2025-06-06T09:54:19Z)
• MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models [10.623996218106564]
  Mixture of Experts (MoE) has become a key architectural paradigm for efficiently scaling Large Language Models (LLMs)<n>We introduce MoLAE, a novel parameterization that reformulating expert operations through a shared projection into a lower-dimensional latent space, followed by expert-specific transformations.<n>We show that MoLAE significantly improves efficiency across multiple dimensions while preserving model capabilities.
  arXiv  Detail & Related papers  (2025-03-29T14:35:34Z)
• Predicting Multi-Agent Specialization via Task Parallelizability [8.465921582175426]
  We present a closed-form bound that predicts when specialization improves performance depending on task regimes and team size.<n>We validate our model on two standard MARL benchmarks that represent opposite benchmarks.<n>Three follow-up experiments in Overcooked-AI demonstrate that the model works in environments with more complex spatial and resource bottlenecks.
  arXiv  Detail & Related papers  (2025-03-19T21:33:48Z)
• Symbolic Mixture-of-Experts: Adaptive Skill-based Routing for Heterogeneous Reasoning [76.10639521319382]
  We propose Symbolic-MoE, a symbolic, text-based, and gradient-free Mixture-of-Experts framework.<n>We show Symbolic-MoE beats strong LLMs like GPT4o-mini, as well as multi-agent approaches, with an absolute avg. gain of 8.15% over the best multi-agent baseline.
  arXiv  Detail & Related papers  (2025-03-07T18:03:13Z)
• Retraining-Free Merging of Sparse MoE via Hierarchical Clustering [24.28646376876676]
  This paper introduces Hierarchical Clustering for Sparsely activated Mixture of Experts (HC-SMoE)<n>HC-SMoE is a task-agnostic expert merging framework for parameter reduction without retraining.<n>We provide theoretical analysis and evaluations across multiple zero-shot language tasks to demonstrate HC-SMoE's effectiveness in state-of-the-art models including Qwen and Mixtral.
  arXiv  Detail & Related papers  (2024-10-11T07:36:14Z)
• MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts [63.67734699877724]
  MoE++ is a general and heterogeneous MoE framework that integrates both Feed-Forward Network(FFN) and zero-computation experts. MoE++ achieves better performance while delivering 1.1-2.1x expert forward throughput compared to a vanilla MoE model of the same size.
  arXiv  Detail & Related papers  (2024-10-09T18:01:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.