Statistic-Augmented, Decoupled MoE Routing and Aggregating in Autonomous Driving

• URL: http://arxiv.org/abs/2512.06664v1
• Date: Sun, 07 Dec 2025 05:28:40 GMT
• Title: Statistic-Augmented, Decoupled MoE Routing and Aggregating in Autonomous Driving
• Abstract summary: Large Model (LM)-Driven Mixture of Experts (MoE) paradigm offers a promising solution.<n>We propose a statistic-augmented, decoupled MoE outing and Aggregating Mechanism (MoE-RAM) driven by LM.<n>MoE-RAM enhances expert routing by incorporating statistical retrieval mechanism to match LM-extracted latent features with cached features of the most relevant experts.
• Score: 32.600054594223096
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Autonomous driving (AD) scenarios are inherently complex and diverse, posing significant challenges for a single deep learning model to effectively cover all possible conditions, such as varying weather, traffic densities, and road types. Large Model (LM)-Driven Mixture of Experts (MoE) paradigm offers a promising solution, where LM serves as the backbone to extract latent features while MoE serves as the downstream head to dynamically select and aggregate specialized experts to adapt to different scenarios. However, routing and aggregating in MoE face intrinsic challenges, including imprecise expert selection due to flawed routing strategy and inefficient expert aggregation leading to suboptimal prediction. To address these issues, we propose a statistic-augmented, decoupled MoE }outing and Aggregating Mechanism (MoE-RAM) driven by LM. Specifically, on the one hand, MoE-RAM enhances expert routing by incorporating statistical retrieval mechanism to match LM-extracted latent features with cached prototypical features of the most relevant experts; on the other hand, MoE-RAM adaptively reweights experts' outputs in fusion by measuring statistical distances of experts' instant features against LM-extracted latent features. Benefiting from the synergy of the statistic-augmented MoE's routing and aggregating, MoE-RAM ultimately improves the prediction performance. We take the AD semantic segmentation task as an example to assess the proposed MoE-RAM. Extensive experiments on AD datasets demonstrate the superiority of MoE-RAM compared to other MoE baselines and conventional single-model approaches.

Related papers

• A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference [10.085075886001665]
  Mixture-of-Experts (MoE) based large language models (LLMs) have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost.<n>We propose ST-MoE, atemporal expert prefetching framework that proactively stages experts ahead of use to overlap expert loading with ongoing computation.
  arXiv  Detail & Related papers  (2026-06-13T20:09:54Z)
• DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts [56.175408382091796]
  We show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router.<n>We propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts.
  arXiv  Detail & Related papers  (2026-05-31T07:08:16Z)
• HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Experts [26.55877320740609]
  We propose HFedMoE, a heterogeneous MoE-based FL fine-tuning framework that customizes a subset of experts to each client.<n> HFedMoE identifies the expert importance based on its contributions to fine-tuning performance.<n>It then adaptively selects a subset of experts from an information bottleneck perspective to align with each client's computing budget.
  arXiv  Detail & Related papers  (2026-01-02T05:56:11Z)
• Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization [60.309915093470416]
  Matryoshka MoE (M-MoE) is a training framework that instills a coarse-to-fine structure directly into the expert ensemble.<n>Our work paves the way for more practical and adaptable deployments of large-scale MoE models.
  arXiv  Detail & Related papers  (2025-09-30T16:56:44Z)
• Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms [55.1784306456972]
  Mixture-of-Experts (MoE) architectures have emerged as a promising direction, offering efficiency and scalability by activating only a subset of parameters during inference.<n>We use an internal metric to investigate the mechanisms of MoE architecture by explicitly incorporating routing mechanisms and analyzing expert-level behaviors.<n>We uncover several findings: (1) neuron utilization decreases as models evolve, reflecting stronger generalization; (2) training exhibits a dynamic trajectory, where benchmark performance alone provides limited signal; (3) task completion emerges from collaborative contributions of multiple experts, with shared experts driving concentration; and (4) activation patterns at the neuron level provide a fine-grained proxy for data diversity.
  arXiv  Detail & Related papers  (2025-09-28T15:13:38Z)
• EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models [25.12002287083368]
  Multi-modal large language models (MLLMs) have increasingly adopted MoE techniques.<n>MoE experts are often by simply replicating the FFN parameters from LLMs.<n>Expert uniformity occurs because MoE experts are often by simply replicating the FFN parameters from LLMs.<n> router rigidity stems from the prevalent use of static linear routers for expert selection.
  arXiv  Detail & Related papers  (2025-05-28T08:38:39Z)
• CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning [10.215751315734018]
  We propose Contrastive Representation for MoE (CoMoE) to promote modularization and specialization in MoE.<n>Experiments on several benchmarks and in multi-task settings demonstrate that CoMoE can consistently enhance MoE's capacity and promote modularization among the experts.
  arXiv  Detail & Related papers  (2025-05-23T06:58:44Z)
• DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism [5.988126768890861]
  DynMoLE is a hybrid routing strategy that dynamically adjusts expert selection based on the Tsallis entropy of the router's probability distribution.<n>Our experiments on commonsense reasoning benchmarks demonstrate that DynMoLE achieves substantial performance improvements.
  arXiv  Detail & Related papers  (2025-04-01T11:14:19Z)
• Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging [111.8456671452411]
  Multi-task learning (MTL) leverages a shared model to accomplish multiple tasks and facilitate knowledge transfer. We propose a Weight-Ensembling Mixture of Experts (WEMoE) method for multi-task model merging. We show that WEMoE and E-WEMoE outperform state-of-the-art (SOTA) model merging methods in terms of MTL performance, generalization, and robustness.
  arXiv  Detail & Related papers  (2024-10-29T07:16:31Z)
• Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts [49.950419707905944]
  We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts. Our approach leverages self-specialization, which constructs expert modules using self-generated synthetic data. Our findings highlight the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.
  arXiv  Detail & Related papers  (2024-06-17T19:06:54Z)
• Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study [65.11303133775857]
  Mixture-of-Experts (MoE) computation amalgamates predictions from several specialized sub-models (referred to as experts) Sparse MoE selectively engages only a limited number, or even just one expert, significantly reducing overhead while empirically preserving, and sometimes even enhancing, performance.
  arXiv  Detail & Related papers  (2024-03-26T05:48:02Z)
• On Least Square Estimation in Softmax Gating Mixture of Experts [78.3687645289918]
  We investigate the performance of the least squares estimators (LSE) under a deterministic MoE model. We establish a condition called strong identifiability to characterize the convergence behavior of various types of expert functions. Our findings have important practical implications for expert selection.
  arXiv  Detail & Related papers  (2024-02-05T12:31:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.