SEAL: Searching Expandable Architectures for Incremental Learning

• URL: http://arxiv.org/abs/2505.10457v2
• Date: Mon, 28 Jul 2025 15:36:46 GMT
• Title: SEAL: Searching Expandable Architectures for Incremental Learning
• Authors: Matteo Gambella, Manuel Roveri,
• Abstract summary: Incremental learning is a machine learning paradigm where a model learns from a sequential stream of tasks.<n>NAS-based approaches to incremental learning often rely on expanding the model at every task, making them impractical in resource-constrained environments.<n>We introduce SEAL, a NAS-based framework tailored for data-incremental learning.<n>SEAL adapts the model structure dynamically by expanding it only when necessary, based on a capacity estimation metric.
• Score: 2.8928489670253277
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Incremental learning is a machine learning paradigm where a model learns from a sequential stream of tasks. This setting poses a key challenge: balancing plasticity (learning new tasks) and stability (preserving past knowledge). Neural Architecture Search (NAS), a branch of AutoML, automates the design of the architecture of Deep Neural Networks and has shown success in static settings. However, existing NAS-based approaches to incremental learning often rely on expanding the model at every task, making them impractical in resource-constrained environments. In this work, we introduce SEAL, a NAS-based framework tailored for data-incremental learning, a scenario where disjoint data samples arrive sequentially and are not stored for future access. SEAL adapts the model structure dynamically by expanding it only when necessary, based on a capacity estimation metric. Stability is preserved through cross-distillation training after each expansion step. The NAS component jointly searches for both the architecture and the optimal expansion policy. Experiments across multiple benchmarks demonstrate that SEAL effectively reduces forgetting and enhances accuracy while maintaining a lower model size compared to prior methods. These results highlight the promise of combining NAS and selective expansion for efficient, adaptive learning in incremental scenarios.

Related papers

• DANCE: Resource-Efficient Neural Architecture Search with Data-Aware and Continuous Adaptation [33.08911251924756]
  We propose DANCE (Dynamic Architectures with Neural Continuous Evolution), which reformulates architecture search as a continuous evolution problem.<n>DANCE introduces three key innovations: a continuous architecture distribution enabling smooth adaptation, a unified architecture space with learned selection gates for efficient sampling, and a multi-stage training strategy for effective deployment optimization.<n>Our method consistently outperforms state-of-the-art NAS approaches in terms of accuracy while significantly reducing search costs.
  arXiv  Detail & Related papers  (2025-07-07T05:22:55Z)
• Federated Neural Architecture Search with Model-Agnostic Meta Learning [7.542593703407386]
  Federated Neural Architecture Search (NAS) enables collaborative search for optimal model architectures tailored to heterogeneous data to achieve higher accuracy.<n>We introduce FedMetaNAS, a framework that integrates meta-learning with NAS within the Federated Learning context.<n>We show that FedMetaNAS significantly accelerates the search process by more than 50% with higher accuracy compared to FedNAS.
  arXiv  Detail & Related papers  (2025-04-08T21:57:40Z)
• Dynamic Allocation Hypernetwork with Adaptive Model Recalibration for Federated Continual Learning [49.508844889242425]
  We propose a novel server-side FCL pattern in medical domain, Dynamic Allocation Hypernetwork with adaptive model recalibration (FedDAH)<n>FedDAH is designed to facilitate collaborative learning under the distinct and dynamic task streams across clients.<n>For the biased optimization, we introduce a novel adaptive model recalibration (AMR) to incorporate the candidate changes of historical models into current server updates.
  arXiv  Detail & Related papers  (2025-03-25T00:17:47Z)
• Dynamic Allocation Hypernetwork with Adaptive Model Recalibration for FCL [49.508844889242425]
  We propose a novel server-side FCL pattern in medical domain, Dynamic Allocation Hypernetwork with adaptive model recalibration (textbfFedDAH)<n>For the biased optimization, we introduce a novel adaptive model recalibration (AMR) to incorporate the candidate changes of historical models into current server updates.<n>Experiments on the AMOS dataset demonstrate the superiority of our FedDAH to other FCL methods on sites with different task streams.
  arXiv  Detail & Related papers  (2025-03-23T13:12:56Z)
• Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning [115.79349923044663]
  Few-shot class-incremental learning (FSCIL) aims to incrementally learn novel classes from limited examples.<n>Existing methods face a critical dilemma: static architectures rely on a fixed parameter space to learn from data that arrive sequentially, prone to overfitting to the current session.<n>In this study, we explore the potential of Selective State Space Models (SSMs) for FSCIL.
  arXiv  Detail & Related papers  (2024-07-08T17:09:39Z)
• Fair Differentiable Neural Network Architecture Search for Long-Tailed Data with Self-Supervised Learning [0.0]
  This paper explores to improve the searching and training performance of NAS on long-tailed datasets. We first discuss the related works about NAS and the deep learning method for long-tailed datasets. Then, we focus on an existing work, called SSF-NAS, which integrates the self-supervised learning and fair differentiable NAS. Finally, we conducted a series of experiments on the CIFAR10-LT dataset for performance evaluation.
  arXiv  Detail & Related papers  (2024-06-19T12:39:02Z)
• Learning Objective-Specific Active Learning Strategies with Attentive Neural Processes [72.75421975804132]
  Learning Active Learning (LAL) suggests to learn the active learning strategy itself, allowing it to adapt to the given setting. We propose a novel LAL method for classification that exploits symmetry and independence properties of the active learning problem. Our approach is based on learning from a myopic oracle, which gives our model the ability to adapt to non-standard objectives.
  arXiv  Detail & Related papers  (2023-09-11T14:16:37Z)
• NASiam: Efficient Representation Learning using Neural Architecture Search for Siamese Networks [76.8112416450677]
  Siamese networks are one of the most trending methods to achieve self-supervised visual representation learning (SSL) NASiam is a novel approach that uses for the first time differentiable NAS to improve the multilayer perceptron projector and predictor (encoder/predictor pair) NASiam reaches competitive performance in both small-scale (i.e., CIFAR-10/CIFAR-100) and large-scale (i.e., ImageNet) image classification datasets while costing only a few GPU hours.
  arXiv  Detail & Related papers  (2023-01-31T19:48:37Z)
• Generalization Properties of NAS under Activation and Skip Connection Search [66.8386847112332]
  We study the generalization properties of Neural Architecture Search (NAS) under a unifying framework. We derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime. We show how the derived results can guide NAS to select the top-performing architectures, even in the case without training.
  arXiv  Detail & Related papers  (2022-09-15T12:11:41Z)
• Towards Self-supervised and Weight-preserving Neural Architecture Search [38.497608743382145]
  We propose the self-supervised and weight-preserving neural architecture search (SSWP-NAS) as an extension of the current NAS framework. Experiments show that the architectures searched by the proposed framework achieve state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet datasets.
  arXiv  Detail & Related papers  (2022-06-08T18:48:05Z)
• Learning to Continuously Optimize Wireless Resource In Episodically Dynamic Environment [55.91291559442884]
  This work develops a methodology that enables data-driven methods to continuously learn and optimize in a dynamic environment. We propose to build the notion of continual learning into the modeling process of learning wireless systems. Our design is based on a novel min-max formulation which ensures certain fairness" across different data samples.
  arXiv  Detail & Related papers  (2020-11-16T08:24:34Z)
• SpaceNet: Make Free Space For Continual Learning [15.914199054779438]
  We propose a novel architectural-based method referred as SpaceNet for class incremental learning scenario. SpaceNet trains sparse deep neural networks from scratch in an adaptive way that compresses the sparse connections of each task in a compact number of neurons. Experimental results show the robustness of our proposed method against catastrophic forgetting old tasks and the efficiency of SpaceNet in utilizing the available capacity of the model.
  arXiv  Detail & Related papers  (2020-07-15T11:21:31Z)
• Learning Architectures from an Extended Search Space for Language Modeling [37.79977691127229]
  We present a general approach to learn both intra-cell and inter-cell architectures of Neural architecture search (NAS) For recurrent neural language modeling, it outperforms a strong baseline significantly on the PTB and WikiText data, with a new state-of-the-art on PTB. The learned architectures show good transferability to other systems.
  arXiv  Detail & Related papers  (2020-05-06T05:02:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.