Drift-aware Collaborative Assistance Mixture of Experts for Heterogeneous Multistream Learning

• URL: http://arxiv.org/abs/2508.01598v1
• Date: Sun, 03 Aug 2025 05:35:34 GMT
• Title: Drift-aware Collaborative Assistance Mixture of Experts for Heterogeneous Multistream Learning
• Abstract summary: Learning from multiple data streams in real-world scenarios is fundamentally challenging due to intrinsic heterogeneity and unpredictable concept drifts.<n>Existing methods typically assume homogeneous streams and employ static architectures with indiscriminate knowledge fusion.<n>We propose CAMEL, a framework that assigns each stream an independent system with a dedicated feature extractor and task-specific head.<n>Furthermore, we propose an Autonomous Expert Tuner (AET) strategy, which dynamically manages expert lifecycles in response to drift.
• Score: 31.877595633244734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning from multiple data streams in real-world scenarios is fundamentally challenging due to intrinsic heterogeneity and unpredictable concept drifts. Existing methods typically assume homogeneous streams and employ static architectures with indiscriminate knowledge fusion, limiting generalizability in complex dynamic environments. To tackle this gap, we propose CAMEL, a dynamic \textbf{C}ollaborative \textbf{A}ssistance \textbf{M}ixture of \textbf{E}xperts \textbf{L}earning framework. It addresses heterogeneity by assigning each stream an independent system with a dedicated feature extractor and task-specific head. Meanwhile, a dynamic pool of specialized private experts captures stream-specific idiosyncratic patterns. Crucially, collaboration across these heterogeneous streams is enabled by a dedicated assistance expert. This expert employs a multi-head attention mechanism to distill and integrate relevant context autonomously from all other concurrent streams. It facilitates targeted knowledge transfer while inherently mitigating negative transfer from irrelevant sources. Furthermore, we propose an Autonomous Expert Tuner (AET) strategy, which dynamically manages expert lifecycles in response to drift. It instantiates new experts for emerging concepts (freezing prior ones to prevent catastrophic forgetting) and prunes obsolete ones. This expert-level plasticity provides a robust and efficient mechanism for online model capacity adaptation. Extensive experiments demonstrate CAMEL's superior generalizability across diverse multistreams and exceptional resilience against complex concept drifts.

Related papers

• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• Prototype-Anchored Generalized Manifold Regression for Unknown-Domain Object Detection [58.25418970608328]
  We study Single-Domain Generalized Object Detection (Single-DGOD), which aims to transfer a detector trained on a single source domain to multiple unseen domains.<n>Existing methods mainly rely on simulation-driven strategies, such as data augmentation or textual prompts, to enlarge the training distribution.<n>We propose Manifold Regression with Visual-Text Dual Chain-of-Thought (MR-DCoT), which formulates unknown-domain generalization as a manifold regression problem.
  arXiv  Detail & Related papers  (2026-07-08T09:27:15Z)
• CoRDE: Concept-Prior Routed Diffusion Experts for Structural Generalization in Robot Manipulation [13.845777652673767]
  Concept-prior Routed Diffusion Experts (CoRDE) is a structure-guided variational distillation framework.<n>CoRDE extracts semantic distributions from a frozen concept encoder to guide the variational posterior responsibility.<n>It forms robust, semantically aligned expert allocations while achieving superior action quality and incremental learning efficiency.
  arXiv  Detail & Related papers  (2026-06-20T08:07:59Z)
• Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution [33.44708543696699]
  MADE-IT orchestrates expert management and activation by grounding intrinsic expert representations in manifold geometry.<n>Experiments demonstrate that MADE-IT consistently outperforms strong baselines in accuracy and robustness across long-horizon and shuffled task sequences.
  arXiv  Detail & Related papers  (2026-04-24T11:35:53Z)
• Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding [45.254713953182716]
  We present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture.<n> Modality-Aware Expert Disentanglement allows shared experts to absorb fine-grained visual semantics from generative tasks.<n>Experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy.
  arXiv  Detail & Related papers  (2026-04-09T03:19:26Z)
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning [83.66308307152808]
  We propose StAbilized Mixture-of-Experts (SAME) for Multimodal Continual Instruction Tuning (MCIT)<n>SAME stabilizes expert selection by decomposing routing dynamics into subspaces and updating only task-relevant directions.<n>It also introduces adaptive expert activation to freeze selected experts during training, reducing redundant and cross-task interference.
  arXiv  Detail & Related papers  (2026-02-02T11:47:06Z)
• HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation [72.69742127579508]
  Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models)<n>In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors.<n> Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge.
  arXiv  Detail & Related papers  (2025-11-25T17:23:38Z)
• Mixture of Ranks with Degradation-Aware Routing for One-Step Real-World Image Super-Resolution [76.66229730098759]
  In real-world image super-resolution (Real-ISR), existing approaches mainly rely on fine-tuning pre-trained diffusion models.<n>We propose a Mixture-of-Ranks (MoR) architecture for single-step image super-resolution.<n>We introduce a fine-grained expert partitioning strategy that treats each rank in LoRA as an independent expert.
  arXiv  Detail & Related papers  (2025-11-20T04:11:44Z)
• GMoPE:A Prompt-Expert Mixture Framework for Graph Foundation Models [30.023472202549076]
  Graph Neural Networks (GNNs) have demonstrated impressive performance on task-specific benchmarks, yet their ability to generalize across diverse domains and tasks remains limited.<n>We propose GMoPE, a framework that seamlessly integrates the Mixture-of-Experts (MoE) architecture with prompt-based learning for graphs.<n>We show that GMoPE consistently outperforms state-of-the-art baselines and achieves performance comparable to full parameter fine-tuning.
  arXiv  Detail & Related papers  (2025-11-05T07:28:51Z)
• RainDiff: End-to-end Precipitation Nowcasting Via Token-wise Attention Diffusion [64.49056527678606]
  We propose a Token-wise Attention integrated into not only the U-Net diffusion model but also the radar-temporal encoder.<n>Unlike prior approaches, our method integrates attention into the architecture without incurring the high resource cost typical of pixel-space diffusion.<n>Our experiments and evaluations demonstrate that the proposed method significantly outperforms state-of-the-art approaches, robustness local fidelity, generalization, and superior in complex precipitation forecasting scenarios.
  arXiv  Detail & Related papers  (2025-10-16T17:59:13Z)
• A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routing [43.930171390617055]
  This paper presents KDP, a knowledge-driven diffusion policy that integrates generative diffusion modeling with a sparse mixture-of-experts routing mechanism.<n>Experiments across representative driving scenarios demonstrate that KDP achieves consistently higher success rates, reduced collision risk, and smoother control compared to prevailing paradigms.<n>These results establish diffusion with expert routing as a scalable and interpretable paradigm for knowledge-driven end-to-end autonomous driving.
  arXiv  Detail & Related papers  (2025-09-05T07:07:18Z)
• DriftMoE: A Mixture of Experts Approach to Handle Concept Drifts [1.2487037582320804]
  This paper introduces DriftMoE, an online Mixture-of-Experts (MoE) architecture that addresses limitations through a novel co-training framework.<n> DriftMoE features a compact neural router that is co-trained alongside a pool of incremental Hoeffding tree experts.<n>We evaluate DriftMoE's performance across nine state-of-the-art data stream learning benchmarks spanning abrupt, gradual, and real-world drifts.
  arXiv  Detail & Related papers  (2025-07-24T14:39:20Z)
• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [50.438552588818]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• Hecto: Modular Sparse Experts for Adaptive and Interpretable Reasoning [0.0]
  Hecto is a lightweight MoE architecture that combines a GRU expert for temporal reasoning and an FFNN expert for static abstraction under a sparse Top-1 gating mechanism.<n>Hecto matches or closely trails homogeneous baselines in performance despite receiving isolated input representations.<n>Hecto establishes itself as a new benchmark for conditional computation.
  arXiv  Detail & Related papers  (2025-06-28T15:03:43Z)
• Cooperation of Experts: Fusing Heterogeneous Information with Large Margin [11.522412489437702]
  Cooperation of Experts (CoE) framework encodes multi-typed information into unified heterogeneous multiplex networks.<n>In our framework, dedicated encoders act as domain-specific experts, each specializing in learning distinct relational patterns in specific semantic spaces.
  arXiv  Detail & Related papers  (2025-05-27T08:04:32Z)
• CoCoAFusE: Beyond Mixtures of Experts via Model Fusion [3.501882879116058]
  CoCoAFusE builds on the philosophy behind Mixtures of Experts (MoEs)<n>Our formulation extends that of a classical Mixture of Experts by contemplating the fusion of the experts' distributions.<n>This new approach is showcased extensively on a suite of motivating numerical examples and a collection of real-data ones.
  arXiv  Detail & Related papers  (2025-05-02T08:35:04Z)
• An Efficient and Mixed Heterogeneous Model for Image Restoration [71.85124734060665]
  Current mainstream approaches are based on three architectural paradigms: CNNs, Transformers, and Mambas.<n>We propose RestorMixer, an efficient and general-purpose IR model based on mixed-architecture fusion.
  arXiv  Detail & Related papers  (2025-04-15T08:19:12Z)
• Deep Reinforcement Learning with Hybrid Intrinsic Reward Model [50.53705050673944]
  Intrinsic reward shaping has emerged as a prevalent approach to solving hard-exploration and sparse-rewards environments.<n>We introduce HIRE (Hybrid Intrinsic REward), a framework for creating hybrid intrinsic rewards through deliberate fusion strategies.
  arXiv  Detail & Related papers  (2025-01-22T04:22:13Z)
• Complexity Experts are Task-Discriminative Learners for Any Image Restoration [80.46313715427928]
  We introduce complexity experts" -- flexible expert blocks with varying computational complexity and receptive fields.<n>This preference effectively drives task-specific allocation, assigning tasks to experts with the appropriate complexity.<n>The proposed MoCE-IR model outperforms state-of-the-art methods, affirming its efficiency and practical applicability.
  arXiv  Detail & Related papers  (2024-11-27T15:58:07Z)
• DeepInteraction++: Multi-Modality Interaction for Autonomous Driving [80.8837864849534]
  We introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout.<n>DeepInteraction++ is a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder.<n>Experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:04:21Z)
• Flexible and Adaptable Summarization via Expertise Separation [59.26639426529827]
  A proficient summarization model should exhibit both flexibility and adaptability. We propose MoeSumm, a Mixture-of-Expert Summarization architecture. Our model's distinct separation of general and domain-specific summarization abilities grants it with notable flexibility and adaptability.
  arXiv  Detail & Related papers  (2024-06-08T05:31:19Z)
• Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study [65.11303133775857]
  Mixture-of-Experts (MoE) computation amalgamates predictions from several specialized sub-models (referred to as experts) Sparse MoE selectively engages only a limited number, or even just one expert, significantly reducing overhead while empirically preserving, and sometimes even enhancing, performance.
  arXiv  Detail & Related papers  (2024-03-26T05:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.