Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation

• URL: http://arxiv.org/abs/2509.16882v1
• Date: Sun, 21 Sep 2025 02:30:04 GMT
• Title: Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
• Abstract summary: We propose DES-MoE, a dynamic expert specialization framework for multi-domain adaptation of Mixture-of-Experts models.<n> DES-MoE addresses catastrophic forgetting through three innovations: (1) an adaptive router balancing pre-trained knowledge retention and task-specific updates via distillation, (2) real-time expert-domain correlation mapping, and (3) a three-phase adaptive fine-tuning schedule that progressively freezes non-specialized parameters.
• Score: 35.38408768980583
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) models offer immense capacity via sparsely gated expert subnetworks, yet adapting them to multiple domains without catastrophic forgetting remains an open challenge. Existing approaches either incur prohibitive computation, suffer cross-domain interference, or require separate runs per domain. We propose DES-MoE, a dynamic expert specialization framework for multi-domain adaptation of Mixture-of-Experts models. DES-MoE addresses catastrophic forgetting through three innovations: (1) an adaptive router balancing pre-trained knowledge retention and task-specific updates via distillation, (2) real-time expert-domain correlation mapping to isolate domain-specific gradients, and (3) a three-phase adaptive fine-tuning schedule that progressively freezes non-specialized parameters. Evaluated on six domains (math, code, law, etc.), DES-MoE matches single-domain ESFT performance while training one unified model, reduces forgetting by 89% compared to full fine-tuning as domains scale from 2 to 6, and achieves 68% faster convergence than conventional methods. Our work establishes dynamic expert isolation as a scalable paradigm for multi-task MoE adaptation.

Related papers

• DSFNet: Learning Dual-Domain Spectral Operators for Multi-Modality Spatio-Temporal Forecasting in Urban Transportation Systems [51.72074484150301]
  We propose the Dual-Domain Spectral Network (DSFNet) to explicitly model the relationships between variables.<n>Our framework employs dual-domain spectral filtering to capture heterogeneous spatial patterns.<n>Compared with the second-best baselines, DSFNet reduces MAE by 3.21%-10.16% across these datasets.
  arXiv  Detail & Related papers  (2026-06-05T07:33:57Z)
• DEER: Disentangled Mixture of Experts with Instance-Adaptive Routing for Generalizable Machine-Generated Text Detection [21.449323711668487]
  We propose a novel framework designed to capture both domain-specific and domain-general MGT patterns.<n>We introduce a mixture-of-experts module, in which domain-specific experts learn fine-grained, domain-local distinctions between human and machine-generated text.<n>We also design a reinforcement learning-based routing mechanism that dynamically selects the appropriate experts for each input instance.
  arXiv  Detail & Related papers  (2025-11-03T03:36:48Z)
• MoE-MLoRA for Multi-Domain CTR Prediction: Efficient Adaptation with Expert Specialization [0.0]
  MoE-MLoRA is a mixture-of-experts framework where each expert is first trained independently to specialize in its domain.<n>We evaluate MoE-MLoRA across eight CTR models on Movielens and Taobao.
  arXiv  Detail & Related papers  (2025-06-09T09:03:05Z)
• TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts [4.5558042369389105]
  TT-LoRA MoE decomposes training into two distinct optimized stages.<n>First, we independently train lightweight, tensorized low-rank adapters (TT-LoRA experts)<n>Subsequently, these expert adapters remain frozen, eliminating inter-task interference and forgetting in multi-task setting.<n>A sparse MoE router, trained separately, dynamically leverages base model representations to select exactly one specialized adapter per input at inference time.<n> Comprehensive experiments confirm our architecture retains the memory efficiency of low-rank adapters, seamlessly scales to large expert pools, and achieves robust task-level optimization.
  arXiv  Detail & Related papers  (2025-04-29T21:46:43Z)
• UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines [64.84631333071728]
  We introduce bfUnistage, a unified Transformer-based framework fortemporal modeling.<n>Our work demonstrates that a task-specific vision-text can build a generalizable model fortemporal learning.<n>We also introduce a temporal module to incorporate temporal dynamics explicitly.
  arXiv  Detail & Related papers  (2025-03-26T17:33:23Z)
• Scalable Multi-Domain Adaptation of Language Models using Modular Experts [10.393155077703653]
  MoDE is a mixture-of-experts architecture that augments a general PLM with modular, domain-specialized experts. MoDE achieves comparable target performances to full parameter fine-tuning while achieving 1.65% better retention performance.
  arXiv  Detail & Related papers  (2024-10-14T06:02:56Z)
• Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation [59.41178047749177]
  We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training. We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling. We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE.
  arXiv  Detail & Related papers  (2024-07-01T09:45:22Z)
• BECoTTA: Input-dependent Online Blending of Experts for Continual Test-time Adaptation [59.1863462632777]
  Continual Test Time Adaptation (CTTA) is required to adapt efficiently to continuous unseen domains while retaining previously learned knowledge. This paper proposes BECoTTA, an input-dependent and efficient modular framework for CTTA. We validate that our method outperforms multiple CTTA scenarios, including disjoint and gradual domain shits, while only requiring 98% fewer trainable parameters.
  arXiv  Detail & Related papers  (2024-02-13T18:37:53Z)
• HAMUR: Hyper Adapter for Multi-Domain Recommendation [49.87140704564021]
  We propose a novel model Hyper Adapter for Multi-Domain Recommendation (HAMUR)<n>HamUR consists of two components.<n>HamUR implicitly captures shared information among domains and dynamically generates the parameters for the adapter.
  arXiv  Detail & Related papers  (2023-09-12T13:34:33Z)
• Multi-path Neural Networks for On-device Multi-domain Visual Classification [55.281139434736254]
  This paper proposes a novel approach to automatically learn a multi-path network for multi-domain visual classification on mobile devices. The proposed multi-path network is learned from neural architecture search by applying one reinforcement learning controller for each domain to select the best path in the super-network created from a MobileNetV3-like search space. The determined multi-path model selectively shares parameters across domains in shared nodes while keeping domain-specific parameters within non-shared nodes in individual domain paths.
  arXiv  Detail & Related papers  (2020-10-10T05:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.