Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning

• URL: http://arxiv.org/abs/2604.14010v1
• Date: Wed, 15 Apr 2026 15:55:38 GMT
• Title: Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning
• Abstract summary: Supervised Fine-Tuning (SFT) of large language models often suffers from task interference and forgetting.<n>Recent approaches alleviate this issue by isolating task-critical parameters during training.<n>We propose Evolving Isolation (EPI), a fine-tuning framework that adapts isolation decisions based on online estimates of parameter importance.
• Score: 26.480133417682897
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Supervised Fine-Tuning (SFT) of large language models often suffers from task interference and catastrophic forgetting. Recent approaches alleviate this issue by isolating task-critical parameters during training. However, these methods represent a static solution to a dynamic problem, assuming that parameter importance remains fixed once identified. In this work, we empirically demonstrate that parameter importance exhibits temporal drift over the course of training. To address this, we propose Evolving Parameter Isolation (EPI), a fine-tuning framework that adapts isolation decisions based on online estimates of parameter importance. Instead of freezing a fixed subset of parameters, EPI periodically updates isolation masks using gradient-based signals, enabling the model to protect emerging task-critical parameters while releasing outdated ones to recover plasticity. Experiments on diverse multi-task benchmarks demonstrate that EPI consistently reduces interference and forgetting compared to static isolation and standard fine-tuning, while improving overall generalization. Our analysis highlights the necessity of synchronizing isolation mechanisms with the evolving dynamics of learning diverse abilities.

Related papers

• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective [52.693471818837395]
  We argue that PEFT should be assessed through the stability-plasticity dilemma.<n>We introduce PEFT-Arena, a benchmark that measures downstream performance and general capability retention.<n>Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.
  arXiv  Detail & Related papers  (2026-05-27T17:59:51Z)
• Disentangled Latent Dynamics Manifold Fusion for Solving Parameterized PDEs [5.601252185916962]
  Generalizing neural surrogate models across different PDE parameters remains difficult.<n>Existing methods usually cannot handle parameter generalization and temporal extrapolation at the same time.<n>We propose Disentangled Latent Dynamics Manifold Fusion (DLDMF), a physics-informed framework that explicitly separates space, time, and parameters.
  arXiv  Detail & Related papers  (2026-03-13T05:46:01Z)
• DPI: Exploiting Parameter Heterogeneity for Interference-Free Fine-Tuning [11.751530422766836]
  Supervised fine-tuning (SFT) is a crucial step for adapting large language models (LLMs) to downstream tasks.<n>We propose a principled approach to disentangle and isolate task-specific parameter regions.
  arXiv  Detail & Related papers  (2026-01-25T10:30:45Z)
• Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs [49.76354497916853]
  Harmonious Adaptation (HPA) is a post-training framework composed of focusing-based parameter partition, harmonious balanced parameter selection, and parameter adjustment.<n>HPA better maintains high safety and mitigates forgetting than existing baselines.<n> experiments on the CVIT benchmark and safety evaluation datasets demonstrate that HPA better maintains high safety and mitigates forgetting than existing baselines.
  arXiv  Detail & Related papers  (2025-11-25T10:34:51Z)
• Continual Adaptation: Environment-Conditional Parameter Generation for Object Detection in Dynamic Scenarios [54.58186816693791]
  environments constantly change over time and space, posing significant challenges for object detectors trained based on a closed-set assumption.<n>We propose a new mechanism, converting the fine-tuning process to a specific- parameter generation.<n>In particular, we first design a dual-path LoRA-based domain-aware adapter that disentangles features into domain-invariant and domain-specific components.
  arXiv  Detail & Related papers  (2025-06-30T17:14:12Z)
• UP-dROM : Uncertainty-Aware and Parametrised dynamic Reduced-Order Model, application to unsteady flows [27.50487430169627]
  Reduced order models (ROMs) play a critical role in fluid mechanics by providing low-cost predictions.<n>For ROMs to be widely applicable, they must not only generalise well across different regimes, but also provide a measure of confidence in their predictions.<n>We present a nonlinear reduction strategy specifically designed for transient flows.
  arXiv  Detail & Related papers  (2025-03-29T22:17:36Z)
• SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models [26.484208658326857]
  Continual learning aims to incrementally acquire new concepts in data streams while resisting forgetting previous knowledge. With the rise of powerful pre-trained models (PTMs), there is a growing interest in training incremental learning systems.
  arXiv  Detail & Related papers  (2024-11-04T15:34:30Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• ETHER: Efficient Finetuning of Large-Scale Models with Hyperplane Reflections [59.839926875976225]
  We propose the ETHER transformation family, which performs Efficient fineTuning via HypErplane Reflections. In particular, we introduce ETHER and its relaxation ETHER+, which match or outperform existing PEFT methods with significantly fewer parameters.
  arXiv  Detail & Related papers  (2024-05-30T17:26:02Z)
• Switching Autoregressive Low-rank Tensor Models [12.461139675114818]
  We show how to switch autoregressive low-rank tensor (SALT) models. SALT parameterizes the tensor of an ARHMM with a low-rank factorization to control the number of parameters. We prove theoretical and discuss practical connections between SALT, linear dynamical systems, and SLDSs.
  arXiv  Detail & Related papers  (2023-06-05T22:25:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.