TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning

• URL: http://arxiv.org/abs/2605.31025v1
• Date: Fri, 29 May 2026 08:57:06 GMT
• Title: TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning
• Abstract summary: We introduce TRACE, a novel approach for discovering Task-specific paRameters.<n>We conduct experiments across multiple standard benchmarks to demonstrate the superior performance of our proposed method.
• Score: 33.07713704586859
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In real-world deployment, LLMs are often adapted continually across tasks to keep LLMs up-to-date in production, where new fine-tuning should preserve previously learned skills. However, indiscriminately mixing tasks can dilute task specialization, while sequential fine-tuning (full-parameter or low rank adaptation) often causes catastrophic forgetting due to destructive overwriting. Replay-based continual tuning and maintaining separate task-specific adapters can mitigate forgetting, but introduce additional compute, storage, and management overhead. Recognizing the redundancy of LLM parameters for any single task, we reframe continual task adaptation as task-specific parameter discovery via adaptation-aware probing: a short warm-start probe exposes a task's adaptation trace, enabling us to identify and isolate the small subset of parameters essential for each task to mitigate catastrophic forgetting. Building on this view, we introduce TRACE, a novel approach for discovering Task-specific paRameters via Adaptation-aware probing for Continual finE-tuning. We perform a short warm-start fine-tune to derive task-specific core parameters by comparing the warm-started and pre-trained models. Core parameters are identified via two strategies: importance scoring (L$_2$ norm and Fisher Information) and specificity analysis (cosine similarity of parameter updates). In continual fine-tuning settings, only the active task's core parameters are updated while others remain frozen, preserving prior knowledge. We conduct extensive experiments across multiple standard benchmarks to demonstrate the superior performance of our proposed method. Additionally, we validate the generalization of our method through a cross-model and scale transferability study, demonstrating a "small-to-large" paradigm that guides the fine-tuning of large-scale models under resource constraints.

Related papers

• One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning [45.405420567098155]
  We present textbfFACET: task-conditioned textbfFetextbfAture transformation with textbfConditiontextbfEd feature consistextbfTency.<n>Our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs.
  arXiv  Detail & Related papers  (2026-08-31T17:03:45Z)
• Attribution-Guided Continual Learning for Large Language Models [32.34727629126595]
  Large language models (LLMs) often suffer from catastrophic forgetting in continual learning.<n>Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization.<n>We propose an attribution-guided continual fine-tuning framework for LLMs.
  arXiv  Detail & Related papers  (2026-05-06T17:38:13Z)
• DPI: Exploiting Parameter Heterogeneity for Interference-Free Fine-Tuning [11.751530422766836]
  Supervised fine-tuning (SFT) is a crucial step for adapting large language models (LLMs) to downstream tasks.<n>We propose a principled approach to disentangle and isolate task-specific parameter regions.
  arXiv  Detail & Related papers  (2026-01-25T10:30:45Z)
• Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging [62.61159948488935]
  Decomposition, Thresholding, and Scaling (DTS) is an approximation-based personalized merging framework.<n>DTS preserves task-specific information with minimal storage overhead.<n>We extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics.
  arXiv  Detail & Related papers  (2025-12-01T09:47:17Z)
• Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance [13.636389424786854]
  Core parameters from each task are transplanted into a unified backbone.<n>Non-core parameters from different tasks are smoothly integrated via Spherical Linear Interpolation.<n>Experiments on multiple public benchmarks demonstrate that our approach significantly alleviates task interference and forgetting.
  arXiv  Detail & Related papers  (2025-08-29T16:07:33Z)
• LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging [80.17238673443127]
  LiNeS is a post-training editing technique designed to preserve pre-trained generalization while enhancing fine-tuned task performance.<n>LiNeS demonstrates significant improvements in both single-task and multi-task settings across various benchmarks in vision and natural language processing.
  arXiv  Detail & Related papers  (2024-10-22T16:26:05Z)
• Task Difficulty Aware Parameter Allocation & Regularization for Lifelong Learning [20.177260510548535]
  We propose the Allocation & Regularization (PAR), which adaptively select an appropriate strategy for each task from parameter allocation and regularization based on its learning difficulty. Our method is scalable and significantly reduces the model's redundancy while improving the model's performance.
  arXiv  Detail & Related papers  (2023-04-11T15:38:21Z)
• Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
  Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers. Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters. We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
  arXiv  Detail & Related papers  (2022-03-30T23:16:07Z)
• Efficient Continual Adaptation for Generative Adversarial Networks [97.20244383723853]
  We present a continual learning approach for generative adversarial networks (GANs) Our approach is based on learning a set of global and task-specific parameters. We show that the feature-map transformation based approach outperforms state-of-the-art continual GANs methods.
  arXiv  Detail & Related papers  (2021-03-06T05:09:37Z)
• Parameter-Efficient Transfer Learning with Diff Pruning [108.03864629388404]
  diff pruning is a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark.
  arXiv  Detail & Related papers  (2020-12-14T12:34:01Z)
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies [57.27944046925876]
  We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph. Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference. Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter.
  arXiv  Detail & Related papers  (2020-01-01T17:34:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.