Scaling Continual Learning with Bi-Level Routing Mixture-of-Experts

• URL: http://arxiv.org/abs/2602.03473v1
• Date: Tue, 03 Feb 2026 12:45:10 GMT
• Title: Scaling Continual Learning with Bi-Level Routing Mixture-of-Experts
• Abstract summary: Class-incremental learning (CIL) on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years.<n>How to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem.<n>We propose CaRE, a scalable Continual Learner with efficient Bi-Level Routing Mixture-of-Experts (BR-MoE)
• Score: 45.405420567098155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning, especially class-incremental learning (CIL), on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years. However, how to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem. We propose CaRE, a scalable {C}ontinual Le{a}rner with efficient Bi-Level {R}outing Mixture-of-{E}xperts (BR-MoE). The core idea of BR-MoE is a bi-level routing mechanism: a router selection stage that dynamically activates relevant task-specific routers, followed by an expert routing phase that dynamically activates and aggregates experts, aiming to inject discriminative and comprehensive representations into every intermediate network layer. On the other hand, we introduce a challenging evaluation protocol for comprehensively assessing CIL methods across very long task sequences spanning hundreds of tasks. Extensive experiments show that CaRE demonstrates leading performance across a variety of datasets and task settings, including commonly used CIL datasets with classical CIL settings (e.g., 5-20 tasks). To the best of our knowledge, CaRE is the first continual learner that scales to very long task sequences (ranging from 100 to over 300 non-overlapping tasks), while outperforming all baselines by a large margin on such task sequences. Code will be publicly released at https://github.com/LMMMEng/CaRE.git.

Related papers

• DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning [6.097306160389619]
  Dynamic expansion methods for class-incremental learning (CIL) protect task-specific knowledge by growing dedicated tokens or computationworks.<n>We identify two intertwined challenges: cross-task confusion from sequential training on predominantly current-task data, and under-optimized shared representations in the backbone.<n>We propose the Decoupled Representation Dynamic Network (DRDN), which addresses these challenges via two mechanisms.
  arXiv  Detail & Related papers  (2026-07-02T02:57:58Z)
• StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs [9.912407135967836]
  We introduce Streaming CVIT (StrCVIT), where models learn from a stream of data chunks containing a dynamic mixture of tasks.<n>In StrCVIT, a model must simultaneously acquire new abilities, reinforce recurring abilities, and forgetting.<n>We propose StrLoRA, a regularized two-stage expert routing framework.
  arXiv  Detail & Related papers  (2026-05-08T06:16:37Z)
• CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansion [9.808005698482914]
  CLARE is a framework for exemplar-free continual learning with vision-language-action models.<n>We show that CLARE achieves high performance on new tasks without catastrophic forgetting of earlier tasks.
  arXiv  Detail & Related papers  (2026-01-14T14:23:42Z)
• Multi-Level Knowledge Distillation and Dynamic Self-Supervised Learning for Continual Learning [29.379585617313552]
  Class-incremental with repetition (CIR) is a more realistic scenario than the traditional class incremental setup.<n>We propose two components that efficiently use the unlabeled data to ensure the high stability and the plasticity of models trained in CIR setup.<n>Both of our proposed components significantly improve the performance in CIR setup, achieving 2nd place in the CVPR 5th CLVISION Challenge.
  arXiv  Detail & Related papers  (2025-08-18T07:50:20Z)
• Separation and Collaboration: Two-Level Routing Grouped Mixture-of-Experts for Multi-Domain Continual Learning [7.361665112773847]
  We propose a Two-Level Grouped Mixture Routing-of-Experts (TRGE) method to mitigate catastrophic forgetting.<n> TRGE dynamically expands the pre-trained CLIP model, assigning specific expert group for each task.<n>We leverage Multimodal Large Language Models (MLLMs) which own powerful multimodal comprehension capabilities to generate task descriptions and recognize the correct task identifier.
  arXiv  Detail & Related papers  (2025-08-11T08:18:22Z)
• CKAA: Cross-subspace Knowledge Alignment and Aggregation for Robust Continual Learning [80.18781219542016]
  Continual Learning (CL) empowers AI models to continuously learn from sequential task streams.<n>Recent parameter-efficient fine-tuning (PEFT)-based CL methods have garnered increasing attention due to their superior performance.<n>We propose Cross-subspace Knowledge Alignment and Aggregation (CKAA) to enhance robustness against misleading task-ids.
  arXiv  Detail & Related papers  (2025-07-13T03:11:35Z)
• LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Models [21.888139819188105]
  LLaVA-CMoE is a continual learning framework for large language models.<n> Probe-Guided Knowledge Extension mechanism determines when and where new experts should be added.<n>Probabilistic Task Locator assigns each task a dedicated, lightweight router.
  arXiv  Detail & Related papers  (2025-03-27T07:36:11Z)
• One Train for Two Tasks: An Encrypted Traffic Classification Framework Using Supervised Contrastive Learning [18.63871240173137]
  We propose an effective model named a Contrastive Learning Enhanced Temporal Fusion (CLE-TFE) In particular, we utilize supervised contrastive learning to enhance the packet-level and flow-level representations. We also propose cross-level multi-task learning, which simultaneously accomplishes the packet-level and flow-level classification tasks in the same model with one training.
  arXiv  Detail & Related papers  (2024-02-12T09:10:09Z)
• TaE: Task-aware Expandable Representation for Long Tail Class Incremental Learning [42.630413950957795]
  We introduce a novel Task-aware Expandable (TaE) framework to learn diverse representations from each incremental task. TaE achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-02-08T16:37:04Z)
• Active Continual Learning: On Balancing Knowledge Retention and Learnability [43.6658577908349]
  Acquiring new knowledge without forgetting what has been learned in a sequence of tasks is the central focus of continual learning (CL) This paper considers the under-explored problem of active continual learning (ACL) for a sequence of active learning (AL) tasks. We investigate the effectiveness and interplay between several AL and CL algorithms in the domain, class and task-incremental scenarios.
  arXiv  Detail & Related papers  (2023-05-06T04:11:03Z)
• Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism [120.1998866178014]
  We present a flexible framework for continual object detection via pRotOtypical taSk corrElaTion guided gaTingAnism (ROSETTA) Concretely, a unified framework is shared by all tasks while task-aware gates are introduced to automatically select sub-models for specific tasks. Experiments on COCO-VOC, KITTI-Kitchen, class-incremental detection on VOC and sequential learning of four tasks show that ROSETTA yields state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-05-06T07:31:28Z)
• Coarse-to-Fine Q-attention with Learned Path Ranking [95.00518278458908]
  We propose Learned Path Ranking (LPR), a method that accepts an end-effector goal pose, and learns to rank a set of goal-reaching paths. In addition to benchmarking our approach across 16 RLBench tasks, we also learn real-world tasks, tabula rasa, in 10-15 minutes, with only 3 demonstrations.
  arXiv  Detail & Related papers  (2022-04-04T15:23:14Z)
• Improving Generalization in Meta-learning via Task Augmentation [69.83677015207527]
  We propose two task augmentation methods, including MetaMix and Channel Shuffle. Both MetaMix and Channel Shuffle outperform state-of-the-art results by a large margin across many datasets.
  arXiv  Detail & Related papers  (2020-07-26T01:50:42Z)
• iTAML: An Incremental Task-Agnostic Meta-learning Approach [123.10294801296926]
  Humans can continuously learn new knowledge as their experience grows. Previous learning in deep neural networks can quickly fade out when they are trained on a new task. We introduce a novel meta-learning approach that seeks to maintain an equilibrium between all encountered tasks.
  arXiv  Detail & Related papers  (2020-03-25T21:42:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.