Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning

• URL: http://arxiv.org/abs/2605.08311v1
• Date: Fri, 08 May 2026 14:07:32 GMT
• Title: Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning
• Abstract summary: Trajectory Regularized Merging (TRM) is a framework that reformulates the merging phase as an optimization process within an augmented trajectory subspace.<n>Our framework integrates three synergistic objectives including task alignment, prediction consistency, and gradient responsiveness to concurrently preserve merged model's historical stability and re-activate optimization dynamics.
• Score: 57.10440766103372
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model merging provides a compelling paradigm for integrating specialized expertise into a unified multi-task model, a goal that aligns naturally with the sequential knowledge acquisition in continual learning (CL). However, the requirement for preserving diverse forms of previous knowledge conflicts with the storage limitations inherent to CL. In this paper, we systematically analyze existing model merging methods under the constraints of CL. We find that current methods prioritize global alignment, which often leads to the accumulation and amplification of task-specific errors within the continuous data stream; and the vanishing gradients at the onset of subsequent tasks frequently cause optimization to stagnate. These leave the merged model in a suboptimal state at the beginning of the next training phase. To address these challenges, we propose Trajectory Regularized Merging (TRM), a framework that reformulates the merging phase as an optimization process within an augmented trajectory subspace. Our framework integrates three synergistic objectives including task alignment, prediction consistency, and gradient responsiveness to concurrently preserve merged model's historical stability and re-activate optimization dynamics. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple benchmarks.

Related papers

• To Retain or to Adapt? Generalizing Continual Learning [39.95178050304739]
  We argue that in non-stationary environments prioritizing retention can impede real-time adaptation.<n>We formalize CL as an online optimization problem governed by the interaction between environmental and learning dynamics.<n>We propose a new general class of continual learning algorithms, which we call Predictive Continual Learning.
  arXiv  Detail & Related papers  (2026-07-06T20:11:53Z)
• Saliency-Aware Model Merging [47.65036144170475]
  Model merging aims to consolidate multiple task-specific models fine-tuned on different datasets into a unified architecture that performs cross-domain proficiency.<n>Current data-free model merging methods often struggle to scale as they rely on simple parameter-levels that ignore inter-layer dependencies and non-uniform distribution of expertise.<n>This work proposes SA-Merging, which is built upon connectivity-based saliency formulations from structural pruning (e.g., SynFlow) and extends them to the data-free model merging setting.
  arXiv  Detail & Related papers  (2026-05-30T04:00:44Z)
• Unlocking the Potential of Continual Model Merging: An ODE Perspective [0.42970700836450487]
  Continual Model Merging (CMM) enables rapid customization of foundation models by sequentially incorporating task-adapted models without repeated retraining.<n>We propose ODE-driven Merging (ODE-M), a controllable framework that formulates each merge as a trajectory in parameter space rather than a one-step endpoint update.
  arXiv  Detail & Related papers  (2026-05-19T06:03:20Z)
• Continual Learning in Vision-Language Models via Aligned Model Merging [84.47520899851557]
  We present a new perspective based on model merging to maintain stability while still retaining plasticity.<n>To maximize the effectiveness of the merging process, we propose a simple mechanism that promotes learning aligned weights with previous ones.
  arXiv  Detail & Related papers  (2025-05-30T20:52:21Z)
• Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning [57.514786046966265]
  We propose textbfPerturb-and-Merge (P&M), a novel continual learning framework that integrates model merging into the CL paradigm to mitigate forgetting.<n>Our proposed approach achieves state-of-the-art performance on several continual learning benchmark datasets.
  arXiv  Detail & Related papers  (2025-05-28T14:14:19Z)
• BECAME: BayEsian Continual Learning with Adaptive Model MErging [21.642774366793997]
  We introduce a two-stage framework named BECAME, which synergizes the expertise of gradient projection and adaptive merging.<n>Our approach outperforms state-of-the-art CL methods and existing merging strategies.
  arXiv  Detail & Related papers  (2025-04-03T15:07:28Z)
• Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging [75.93960998357812]
  Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their capabilities across different tasks and domains.<n>Current model merging techniques focus on merging all available models simultaneously, with weight matrices-based methods being the predominant approaches.<n>We propose a training-free projection-based continual merging method that processes models sequentially.
  arXiv  Detail & Related papers  (2025-01-16T13:17:24Z)
• Continual Task Learning through Adaptive Policy Self-Composition [54.95680427960524]
  CompoFormer is a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Our experiments reveal that CompoFormer outperforms conventional continual learning (CL) methods, particularly in longer task sequences.
  arXiv  Detail & Related papers  (2024-11-18T08:20:21Z)
• AdaMerging: Adaptive Model Merging for Multi-Task Learning [68.75885518081357]
  This paper introduces an innovative technique called Adaptive Model Merging (AdaMerging) It aims to autonomously learn the coefficients for model merging, either in a task-wise or layer-wise manner, without relying on the original training data. Compared to the current state-of-the-art task arithmetic merging scheme, AdaMerging showcases a remarkable 11% improvement in performance.
  arXiv  Detail & Related papers  (2023-10-04T04:26:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.