Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer

• URL: http://arxiv.org/abs/2605.28444v1
• Date: Wed, 27 May 2026 13:10:34 GMT
• Title: Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer
• Abstract summary: When a new version of a pre-trained model becomes available, expertise acquired through fine-tuning cannot be directly reused.<n>We propose BiCo, a training-free framework for transferring task vectors through Bilinear Coordinate alignment.<n>BiCo consistently outperforms existing transfer methods across models that differ in width, depth, and pre-training configuration.
• Score: 13.823003260600663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning large-scale pre-trained models is a recent prevalent paradigm for adapting general representations to specialized tasks. However, when a new version of a pre-trained model becomes available, expertise acquired through fine-tuning cannot be directly reused because it is tied to the parameterization of the original model, requiring another costly fine-tuning. To address this inefficiency, recent work uses task vectors, defined as the parameter difference between a fine-tuned model and its base model, to transfer expertise across models. While existing methods bridge disparate models by matching activations or gradients, a significant performance gap remains relative to direct fine-tuning, suggesting that these partial correspondences are insufficient. In this work, instead of viewing a task vector merely as a parameter offset, we revisit the formation of task vectors and show that they can be derived as accumulated bilinear interactions between input-side activations and output-side gradients. Motivated by this observation, we formulate task-vector transfer as a dual-space alignment problem and propose BiCo, a training-free framework for transferring task vectors through Bilinear Coordinate alignment. BiCo estimates orthogonal Procrustes mappings in both spaces using a single forward-backward pass on a small calibration set, without any parameter update. Across extensive computer vision and natural language processing benchmarks, BiCo consistently outperforms existing transfer methods across models that differ in width, depth, and pre-training configuration.

Related papers

• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic [17.222346684974607]
  We bridge the gap between linear and standard non-linear fine-tuning.<n>We distill hidden representations from a curvature-regularized linearized teacher into a non-linear student trained via conventional fine-tuning.<n>We find that the resulting model inherits key properties of linearized models for task arithmetic, enabling effective composition of task vectors and achieving strong performance across vision and language benchmarks without incurring any inference-time overhead.
  arXiv  Detail & Related papers  (2026-05-18T18:11:34Z)
• Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models [25.83401080149413]
  We show that the key to successful transfer lies in the sign structure of the gradients of the new model.<n>We propose GradFix, a novel method that approximates the ideal gradient sign structure.<n>We demonstrate significant performance gains on vision and language benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T13:30:25Z)
• Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignment [5.2980803808373516]
  Task arithmetic enables efficient model editing by representing task-specific changes as vectors in parameter space.<n>This assumption limits its applicability in cross-model transfer settings, where models are independently pre-trained on different datasets.<n>We propose a method based on few-shot alignment, which aligns task vectors to the parameter space of a differently pre-trained target model.
  arXiv  Detail & Related papers  (2025-05-17T14:24:06Z)
• When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers [64.1656365676171]
  Task arithmetic refers to editing the pre-trained model by adding a weighted sum of task vectors.<n>This paper theoretically prove the effectiveness of task addition in simultaneously learning a set of irrelevant or irrelevant tasks.<n>We prove the proper selection for task arithmetic to achieve negation to out-of-domain tasks.
  arXiv  Detail & Related papers  (2025-04-15T08:04:39Z)
• Efficient Model Editing with Task-Localized Sparse Fine-tuning [14.792099973449794]
  We propose TaLoS which allows to build sparse task vectors with minimal interference without requiring explicit linearization.<n>We find that pre-trained models contain a subset of parameters with consistently low gradient sensitivity across tasks.<n>Our experiments prove that TaLoS improves training and inference efficiency while outperforming current methods in task addition and negation.
  arXiv  Detail & Related papers  (2025-04-03T14:20:06Z)
• Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging [75.93960998357812]
  Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their capabilities across different tasks and domains.<n>Current model merging techniques focus on merging all available models simultaneously, with weight matrices-based methods being the predominant approaches.<n>We propose a training-free projection-based continual merging method that processes models sequentially.
  arXiv  Detail & Related papers  (2025-01-16T13:17:24Z)
• Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach [87.8330887605381]
  We show how to adapt a pre-trained Vision Transformer to downstream recognition tasks with only a few learnable parameters. We synthesize a task-specific query with a learnable and lightweight module, which is independent of the pre-trained model. Our method achieves state-of-the-art performance under memory constraints, showcasing its applicability in real-world situations.
  arXiv  Detail & Related papers  (2024-07-09T15:45:04Z)
• Pre-Trained Model Recommendation for Downstream Fine-tuning [22.343011779348682]
  Model selection aims to rank off-the-shelf pre-trained models and select the most suitable one for the new target task. Existing model selection techniques are often constrained in their scope and tend to overlook the nuanced relationships between models and tasks. We present a pragmatic framework textbfFennec, delving into a diverse, large-scale model repository.
  arXiv  Detail & Related papers  (2024-03-11T02:24:32Z)
• On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm [47.55215041326702]
  We discover an intriguing linear phenomenon in models that are from a common pretrained checkpoint and finetuned on different tasks, termed as Cross-Task Linearity (CTL) We show that if we linearly interpolate the weights of two finetuned models, the features in the weight-interpolated model are often approximately equal to the linearities of features in two finetuned models at each layer. We conjecture that in the pretraining-finetuning paradigm, neural networks approximately function as linear maps, mapping from the parameter space to the feature space.
  arXiv  Detail & Related papers  (2024-02-06T03:28:36Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving [103.745551954983]
  In this paper, we investigate the transfer performance of various types of self-supervised methods, including MoCo and SimCLR, on three downstream tasks. We find that their performances are sub-optimal or even lag far behind the single-task baseline. We propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training.
  arXiv  Detail & Related papers  (2022-09-19T12:15:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.