Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature

• URL: http://arxiv.org/abs/2602.17385v2
• Date: Mon, 23 Feb 2026 17:36:15 GMT
• Title: Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature
• Abstract summary: We propose a dataless approach by framing regularization against representation drift as a curvature matrix approximation problem.<n>In particular, we adopt Kronecker-Factored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and negation.
• Score: 29.0138600508929
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Task Arithmetic yields a modular, scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors; however, existing approaches typically require external task data, conflicting with modularity and data availability constraints (e.g., privacy requirements). We propose a dataless approach by framing regularization against representation drift as a curvature matrix approximation problem. This allows us to leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and negation. Our method has constant complexity in the number of tasks and promotes robustness to task vector rescaling, eliminating the need for held-out tuning.

Related papers

• OrthoTryOn: Geometric Orthogonalization for Conflict-Free Unified Fashion Generation [43.40052085990979]
  Unified fashion generation integrates tasks like virtual try-on and garment reconstruction into a single model to reduce task-specific adaptation costs.<n>We propose OrthoTryOn, a unified framework mitigating this interference within a shared Low- AdaptationRank (LoRA) module.
  arXiv  Detail & Related papers  (2026-06-26T09:23:25Z)
• Scalable Gaussian process modeling of parametrized spatio-temporal fields [2.005299372367689]
  We develop a scalable framework for learning of parametized equations over fixed or parameter-temporal domains.<n>A key feature of our approach is the efficient computation of the posterior variance at essentially the same computational cost as the posterior mean.<n>Results establish the proposed framework as an effective tool for data-driven surrogate modeling, particularly when uncertainty estimates are required for downstream tasks.
  arXiv  Detail & Related papers  (2026-02-27T20:16:21Z)
• When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers [64.1656365676171]
  Task arithmetic refers to editing the pre-trained model by adding a weighted sum of task vectors.<n>This paper theoretically prove the effectiveness of task addition in simultaneously learning a set of irrelevant or irrelevant tasks.<n>We prove the proper selection for task arithmetic to achieve negation to out-of-domain tasks.
  arXiv  Detail & Related papers  (2025-04-15T08:04:39Z)
• Efficient Model Editing with Task-Localized Sparse Fine-tuning [14.792099973449794]
  We propose TaLoS which allows to build sparse task vectors with minimal interference without requiring explicit linearization.<n>We find that pre-trained models contain a subset of parameters with consistently low gradient sensitivity across tasks.<n>Our experiments prove that TaLoS improves training and inference efficiency while outperforming current methods in task addition and negation.
  arXiv  Detail & Related papers  (2025-04-03T14:20:06Z)
• Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors [27.848233831749216]
  textbfWUDI-Merging (textbfWhoever started the interference shotextbfUld entextbfD textbfIt) is a model merging method that eliminates interference without any additional data or rescaling coefficients.<n> Comprehensive empirical evaluations across vision and language benchmarks demonstrate our method's superiority.
  arXiv  Detail & Related papers  (2025-03-11T07:01:35Z)
• Multi-Task Model Merging via Adaptive Weight Disentanglement [69.7292615212444]
  We introduce an Adaptive Weight Disentanglement method for model merging.<n>We successfully extract redundant vectors, and after their subtraction, the task vectors retain robust performance.
  arXiv  Detail & Related papers  (2024-11-27T20:08:55Z)
• CoRMF: Criticality-Ordered Recurrent Mean Field Ising Solver [4.364088891019632]
  We propose an RNN-based efficient Ising model solver, the Criticality-ordered Recurrent Mean Field (CoRMF) By leveraging the approximated tree structure of the underlying Ising graph, the newly-obtained criticality order enables the unification between variational mean-field and RNN. CoRFM solves the Ising problems in a self-train fashion without data/evidence, and the inference tasks can be executed by directly sampling from RNN.
  arXiv  Detail & Related papers  (2024-03-05T16:55:06Z)
• Toward Certified Robustness Against Real-World Distribution Shifts [65.66374339500025]
  We train a generative model to learn perturbations from data and define specifications with respect to the output of the learned model. A unique challenge arising from this setting is that existing verifiers cannot tightly approximate sigmoid activations. We propose a general meta-algorithm for handling sigmoid activations which leverages classical notions of counter-example-guided abstraction refinement.
  arXiv  Detail & Related papers  (2022-06-08T04:09:13Z)
• High-Dimensional Robust Mean Estimation via Gradient Descent [73.61354272612752]
  We show that the problem of robust mean estimation in the presence of a constant adversarial fraction can be solved by gradient descent. Our work establishes an intriguing connection between the near non-lemma estimation and robust statistics.
  arXiv  Detail & Related papers  (2020-05-04T10:48:04Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)
• Polynomial-Time Exact MAP Inference on Discrete Models with Global Dependencies [83.05591911173332]
  junction tree algorithm is the most general solution for exact MAP inference with run-time guarantees. We propose a new graph transformation technique via node cloning which ensures a run-time for solving our target problem independently of the form of a corresponding clique tree.
  arXiv  Detail & Related papers  (2019-12-27T13:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.