Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution

• URL: http://arxiv.org/abs/2604.22464v1
• Date: Fri, 24 Apr 2026 11:35:53 GMT
• Title: Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution
• Abstract summary: MADE-IT orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry.<n>Experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences.
• Score: 33.44708543696699
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual Model Merging (CMM) sequentially integrates task-specific models into a unified architecture without intensive retraining. However, existing CMM methods are hindered by a fundamental saturation-redundancy dilemma: backbone-centric approaches face parameter saturation and representation interference within fixed capacities, whereas Mixture-of-Experts (MoE) variants resort to indiscriminate expansion, incurring expert redundancy and a routing bottleneck reliant on additional data-driven optimization. To resolve these challenges, we propose MADE-IT (Manifold-Aware Dynamic Expert Evolution and Implicit rouTing), an adaptive CMM method that orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry. We introduce a projection-based subspace affinity metric coupled with a distribution-aware adaptive threshold mechanism to guide autonomous expert evolution, harmonizing diversity with architectural parsimony. Furthermore, to bypass parameterized gating networks, we design a data-free and training-free implicit routing mechanism that activates experts via feature-subspace alignment. Extensive experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences, while significantly pruning redundant experts, particularly within generic modules and early layers.

Related papers

• SuppreSensing: Expert-Guided Feature Recalibration and Discrepancy Augmentation for Multimodal Object Detection [19.446402308279676]
  We propose SuppreSensing, which reformulates multimodal fusion as a selective collaboration process.<n>SupreSensing first designs an Expert-driven Multimodal Feature Recalibration (EMFR) module.<n>A modality-specific augmentation strategy is employed to attribute specific modality features.<n>Experiments on the DroneVehicle and VEDAI datasets demonstrate that SuppreSensing achieves state-of-the-art detection performance.
  arXiv  Detail & Related papers  (2026-08-21T10:06:00Z)
• CoRDE: Concept-Prior Routed Diffusion Experts for Structural Generalization in Robot Manipulation [13.845777652673767]
  Concept-prior Routed Diffusion Experts (CoRDE) is a structure-guided variational distillation framework.<n>CoRDE extracts semantic distributions from a frozen concept encoder to guide the variational posterior responsibility.<n>It forms robust, semantically aligned expert allocations while achieving superior action quality and incremental learning efficiency.
  arXiv  Detail & Related papers  (2026-06-20T08:07:59Z)
• Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models [6.871645384293096]
  Mixture-of-Experts (MoE) models scale foundation models efficiently by activating only a subset of experts for each token.<n>MoE models are sensitive to routing instability: small quantization-induced perturbations can change the top-$k$ expert selection.<n>We propose Value-and-Structure Routing Alignment for Quantization (VSRAQ) to preserve expert-selection behavior under quantization.
  arXiv  Detail & Related papers  (2026-06-04T04:13:05Z)
• AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE [51.994752158953084]
  AnchorMoE is an interpretable-by-construction classification framework.<n>It encodes multi-view representations of local patches and routes them to specialized experts.<n>Experiments on real-world and synthetic benchmarks demonstrate that AnchorMoE achieves highly competitive classification performance.
  arXiv  Detail & Related papers  (2026-06-02T13:30:54Z)
• DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts [56.175408382091796]
  We show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router.<n>We propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts.
  arXiv  Detail & Related papers  (2026-05-31T07:08:16Z)
• From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderation [59.27094165576015]
  We propose a novel learning paradigm (UniMod) that transitions from sparse decision-making to dense reasoning traces.<n>By constructing structured trajectories encompassing evidence grounding, modality assessment, risk mapping, policy decision, and response generation, we reformulate monolithic decision tasks into a multi-dimensional boundary learning process.<n>We introduce specialized optimization strategies to decouple task-specific parameters and rebalance training dynamics, effectively resolving interference between diverse objectives in multi-task learning.
  arXiv  Detail & Related papers  (2026-01-28T09:29:40Z)
• Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures [2.538209532048867]
  Mixture of Experts (MoE) architectures enable efficient scaling of neural networks but suffer from expert collapse.<n>We propose the Spectrally-Regularized Mixture of Experts (SR-MoE), which imposes geometric constraints on the routing manifold to enforce structural modularity.
  arXiv  Detail & Related papers  (2026-01-07T12:59:37Z)
• How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts [30.125087273625123]
  We propose a semanticaware MoE framework for adaptive expert expansion and dynamic routing.<n>MASS converges to the point of optimal balance between cost-performance trade-off and notably improved sematic specialization.
  arXiv  Detail & Related papers  (2025-12-21T05:37:42Z)
• HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation [72.69742127579508]
  Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models)<n>In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors.<n> Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge.
  arXiv  Detail & Related papers  (2025-11-25T17:23:38Z)
• Mixture of Ranks with Degradation-Aware Routing for One-Step Real-World Image Super-Resolution [76.66229730098759]
  In real-world image super-resolution (Real-ISR), existing approaches mainly rely on fine-tuning pre-trained diffusion models.<n>We propose a Mixture-of-Ranks (MoR) architecture for single-step image super-resolution.<n>We introduce a fine-grained expert partitioning strategy that treats each rank in LoRA as an independent expert.
  arXiv  Detail & Related papers  (2025-11-20T04:11:44Z)
• Adapformer: Adaptive Channel Management for Multivariate Time Series Forecasting [49.40321003932633]
  Adapformer is an advanced Transformer-based framework that merges the benefits of CI and CD methodologies through effective channel management.<n>Adapformer achieves superior performance over existing models, enhancing both predictive accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-11-18T16:24:05Z)
• Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms [55.1784306456972]
  Mixture-of-Experts (MoE) architectures have emerged as a promising direction, offering efficiency and scalability by activating only a subset of parameters during inference.<n>We use an internal metric to investigate the mechanisms of MoE architecture by explicitly incorporating routing mechanisms and analyzing expert-level behaviors.<n>We uncover several findings: (1) neuron utilization decreases as models evolve, reflecting stronger generalization; (2) training exhibits a dynamic trajectory, where benchmark performance alone provides limited signal; (3) task completion emerges from collaborative contributions of multiple experts, with shared experts driving concentration; and (4) activation patterns at the neuron level provide a fine-grained proxy for data diversity.
  arXiv  Detail & Related papers  (2025-09-28T15:13:38Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts [10.204413386807564]
  We propose a Dynamic Adaptive Shared Expert and Grouped Multi-Head Attention Hybrid Model (DASG-MoE) to enhance long-sequence modeling capabilities.<n>First, we employ the Grouped Multi-Head Attention (GMHA) mechanism to effectively reduce the computational complexity of long sequences.<n>Second, we design a Dual-Scale Shared Expert Structure (DSSE), where shallow experts use lightweight computations to quickly respond to low-dimensional features.<n>Third, we propose a hierarchical Adaptive Dynamic Routing (ADR) mechanism that dynamically selects expert levels based on feature complexity and task requirements.
  arXiv  Detail & Related papers  (2025-09-05T02:49:15Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• Single Domain Generalization with Model-aware Parametric Batch-wise Mixup [22.709796153794507]
  Single Domain Generalization remains a formidable challenge in the field of machine learning.<n>We propose a novel data augmentation approach, named as Model-aware Parametric Batch-wise Mixup.<n>By exploiting inter-feature correlations, the parameterized mixup generator introduces additional versatility in combining features across a batch of instances.
  arXiv  Detail & Related papers  (2025-02-22T03:45:18Z)
• Retraining-Free Merging of Sparse MoE via Hierarchical Clustering [24.28646376876676]
  This paper introduces Hierarchical Clustering for Sparsely activated Mixture of Experts (HC-SMoE)<n>HC-SMoE is a task-agnostic expert merging framework for parameter reduction without retraining.<n>We provide theoretical analysis and evaluations across multiple zero-shot language tasks to demonstrate HC-SMoE's effectiveness in state-of-the-art models including Qwen and Mixtral.
  arXiv  Detail & Related papers  (2024-10-11T07:36:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.