Per-parameter Task Arithmetic for Unlearning in Large Language Models

• URL: http://arxiv.org/abs/2601.22030v1
• Date: Thu, 29 Jan 2026 17:35:32 GMT
• Title: Per-parameter Task Arithmetic for Unlearning in Large Language Models
• Authors: Chengyi Cai, Zesheng Ye, Jiangchao Yao, Jianzhong Qi, Bo Han, Xiaolu Zhang, Feng Liu, Jun Zhou,
• Abstract summary: In large language model (LLM) unlearning, private information is required to be removed.<n> Task arithmetic unlearns by subtracting a specific task vector (TV)
• Score: 63.240204423180955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In large language model (LLM) unlearning, private information is required to be removed. Task arithmetic unlearns by subtracting a specific task vector (TV)--defined as the parameter difference between a privacy-information-tuned model and the original model. While efficient, it can cause over-forgetting by disrupting parameters essential for retaining other information. Motivated by the observation that each parameter exhibits different importance for forgetting versus retention, we propose a per-parameter task arithmetic (PerTA) mechanism to rescale the TV, allowing per-parameter adjustment. These weights quantify the relative importance of each parameter for forgetting versus retention, estimated via gradients (i.e., PerTA-grad) or the diagonal Fisher information approximation (i.e., PerTA-fisher). Moreover, we discuss the effectiveness of PerTA, extend it to a more general form, and provide further analysis. Extensive experiments demonstrate that PerTA consistently improves upon standard TV, and in many cases surpasses widely used training-based unlearning methods in both forgetting effectiveness and overall model utility. By retaining the efficiency of task arithmetic while mitigating over-forgetting, PerTA offers a principled and practical framework for LLM unlearning.

Related papers

• Learning to Weight Parameters for Training Data Attribution [62.830878652285406]
  We propose a method to explicitly learn parameter importance weights directly from data, without requiring annotated labels.<n>Our approach improves attribution accuracy across diverse tasks, including image classification, language modeling, and diffusion, and enables fine-grained attribution for concepts like subject and style.
  arXiv  Detail & Related papers  (2025-06-06T00:32:04Z)
• Unified Parameter-Efficient Unlearning for LLMs [25.195126838721492]
  Large Language Models (LLMs) have revolutionized natural language processing, enabling advanced understanding and reasoning capabilities across a variety of tasks.<n>This raises significant privacy and security concerns, as models may inadvertently retain and disseminate sensitive or undesirable information.<n>We introduce a novel instance-wise unlearning framework, LLMEraser, which systematically categorizes unlearning tasks and applies precise adjustments using influence functions.
  arXiv  Detail & Related papers  (2024-11-30T07:21:02Z)
• Beyond Task Vectors: Selective Task Arithmetic Based on Importance Metrics [0.0]
  This paper introduces textbfunderlineSelective textbfunderlineTask textbfunderlineArithmetic underlinetextbf(STA), a training-free framework designed to enhance multi-task performance through task-specific parameter fusion. Experimental results demonstrate that STA achieves superior multi-task performance across benchmarks and excellent performance in task forgetting.
  arXiv  Detail & Related papers  (2024-11-25T06:59:16Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding [40.27182770995891]
  Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. We introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech-processing tasks.
  arXiv  Detail & Related papers  (2023-03-02T08:57:33Z)
• Scalable Weight Reparametrization for Efficient Transfer Learning [10.265713480189486]
  Efficient transfer learning involves utilizing a pre-trained model trained on a larger dataset and repurposing it for downstream tasks. Previous works have led to an increase in updated parameters and task-specific modules, resulting in more computations, especially for tiny models. We suggest learning a policy network that can decide where to reparametrize the pre-trained model, while adhering to a given constraint for the number of updated parameters.
  arXiv  Detail & Related papers  (2023-02-26T23:19:11Z)
• Towards a Unified View on Visual Parameter-Efficient Transfer Learning [96.99924127527002]
  We propose a framework with a unified view called visual-PETL (V-PETL) to investigate the different aspects affecting the trade-off. An effective scheme Swin-BAPAT derived from the proposed V-PETL framework achieves significantly better performance than the state-of-the-art AdaptFormer-Swin.
  arXiv  Detail & Related papers  (2022-10-03T09:54:39Z)
• Parameter-Efficient Sparsity for Large Language Models Fine-Tuning [63.321205487234074]
  We propose a. sparse-efficient Sparse Training (PST) method to reduce the number of trainable parameters during sparse-aware training. Experiments with diverse networks (i.e., BERT, RoBERTa and GPT-2) demonstrate PST performs on par or better than previous sparsity methods.
  arXiv  Detail & Related papers  (2022-05-23T02:43:45Z)
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning [81.3514358542452]
  Few-shot in-context learning (ICL) incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made. parameter-efficient fine-tuning offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task. In this paper, we rigorously compare few-shot ICL and parameter-efficient fine-tuning and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs.
  arXiv  Detail & Related papers  (2022-05-11T17:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.