Less is More: Efficient Model Merging with Binary Task Switch

• URL: http://arxiv.org/abs/2412.00054v1
• Date: Sun, 24 Nov 2024 14:47:13 GMT
• Title: Less is More: Efficient Model Merging with Binary Task Switch
• Abstract summary: We introduce Task Switch, which decomposes task vectors into three components.<n>By storing task vectors in a binarized form, T-Switch alleviates parameter conflicts while ensuring efficient task parameter storage.<n>Experiments indicate that our methods achieve significant performance improvements over existing baselines, requiring only 1-3% of the storage space of full-precision parameters.
• Score: 19.622432243113852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As an effective approach to equip models with multi-task capabilities without additional training, model merging has garnered significant attention. However, existing methods face challenges of redundant parameter conflicts and the excessive storage burden of parameters. In this work, through controlled experiments, we reveal that for task vectors, only those parameters with magnitudes above a certain threshold contribute positively to the task, exhibiting a pulse-like characteristic. We then attempt leveraging this characteristic to binarize the task vectors and reduce storage overhead. Further controlled experiments show that the binarized task vectors incur almost no decrease in fine-tuning and merging performance, and even exhibit stronger performance improvements as the proportion of redundant parameters increases. Based on these insights, we propose Task Switch (T-Switch), which decomposes task vectors into three components: 1) an activation switch instantiated by a binarized mask vector, 2) a polarity switch instantiated by a binarized sign vector, and 3) a scaling knob instantiated by a scalar coefficient. By storing task vectors in a binarized form, T-Switch alleviates parameter conflicts while ensuring efficient task parameter storage. Furthermore, to enable automated switch combination in T-Switch, we further introduce Auto-Switch, which enables training-free switch combination via retrieval from a small query set. Experiments indicate that our methods achieve significant performance improvements over existing baselines, requiring only 1-3% of the storage space of full-precision parameters.

Related papers

• T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler [51.05812650668468]
  Multi-task reinforcement learning (MTRL) is a technique to train multiple tasks simultaneously, where previous works usually train a single model to solve different tasks by sharing parameters across various tasks.<n>We propose a novel MTRL framework called Task-Specific feature Selector and Scheduler (T3S), which consists of two components: a feature selector and a task scheduler.<n>Specifically, the feature selectors employ hypernetworks to construct task-specific soft masks, which can be applied by globally shared representation to construct task-specific features.
  arXiv  Detail & Related papers  (2026-08-31T13:30:06Z)
• Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression [54.12624130407679]
  We propose T-Switch, which decomposes task vectors into three compact components.<n>We then introduce Auto-Switch, a training-free merging scheme that automatically composes task vectors.<n>We present Auto-FlexSwitch, a dynamic model merging approach that supports highly efficient task vector compression.
  arXiv  Detail & Related papers  (2026-04-30T16:58:05Z)
• pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training [24.05577787968274]
  pQuant is a method that decouples parameters by splitting linear layers into two specialized branches.<n>We show pQuant achieves state-of-the-art performance in extremely low-bit quantization.
  arXiv  Detail & Related papers  (2026-02-26T03:51:58Z)
• TR-PTS: Task-Relevant Parameter and Token Selection for Efficient Tuning [41.097430916756]
  Large pre-trained models achieve remarkable performance in vision tasks but are impractical for fine-tuning due to high computational and storage costs.<n>We propose Task-Relevant.<n>and Token Selection (TR-PTS), a task-driven framework that enhances both computational efficiency and accuracy.<n>We evaluate TR-PTS on benchmark, including FGVC and VTAB-1k, where it achieves state-of-the-art performance, surpassing full fine-tuning by 3.40% and 10.35%, respectively.
  arXiv  Detail & Related papers  (2025-07-30T17:47:13Z)
• Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformer [56.898822179122476]
  We propose M3DT, a novel mixture-of-experts (MoE) framework that tackles task scalability by further unlocking the model's parameter scalability.<n> Experimental results show that, by increasing the number of experts, M3DT not only consistently enhances its performance as model expansion on the fixed task numbers, but also exhibits remarkable task scalability, successfully extending to 160 tasks with superior performance.
  arXiv  Detail & Related papers  (2025-05-30T09:08:52Z)
• Task Vector Quantization for Memory-Efficient Model Merging [11.777982281994193]
  We propose quantizing task vectors instead of quantizing fine-tuned checkpoints. Our method maintains or improves model merging performance while using only 8% of the memory required for full-precision checkpoints.
  arXiv  Detail & Related papers  (2025-03-10T05:00:24Z)
• Multi-Task Model Merging via Adaptive Weight Disentanglement [69.7292615212444]
  Model merging is a technique for integrating task-specific weights from various tasks into a unified multi-task model without retraining or additional data.<n> Task Arithmetic (TA) has demonstrated that combining task vectors through arithmetic operations facilitates efficient capability transfer between different tasks.<n>Despite the notable effectiveness of TA, interference among task vectors can adversely affect the performance of the merged model.<n>We propose Adaptive Weight Disentanglement (AWD), which decomposes traditional task vectors into a vector redundant and several disentangled task vectors.
  arXiv  Detail & Related papers  (2024-11-27T20:08:55Z)
• Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision [52.80792724919329]
  We introduce a novel framework named Adapter-X to improve fine-tuning in 2D image and 3D point cloud modalities. It is the first to outperform full fine-tuning in both 2D image and 3D point cloud modalities with significantly fewer parameters, i.e., only 0.20% and 1.88% of original trainable parameters for 2D and 3D classification tasks.
  arXiv  Detail & Related papers  (2024-06-05T08:26:44Z)
• Localizing Task Information for Improved Model Merging and Compression [61.16012721460561]
  We show that the information required to solve each task is still preserved after merging as different tasks mostly use non-overlapping sets of weights. We propose Consensus Merging, an algorithm that eliminates such weights and improves the general performance of existing model merging approaches.
  arXiv  Detail & Related papers  (2024-05-13T14:54:37Z)
• ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale [18.396897413970965]
  ScaLearn is a simple and highly parameter-efficient two-stage MTL method. We show that ScaLearn consistently outperforms strong baselines with a small number of transfer parameters.
  arXiv  Detail & Related papers  (2023-10-02T14:01:36Z)
• Prompt Guided Transformer for Multi-Task Dense Prediction [14.815576352301322]
  We introduce a lightweight task-conditional model called Prompt Guided Transformer to optimize performance and model parameters. Our approach achieves state-of-the-art results among task-conditional methods while using fewer parameters, and maintains a significant balance between performance and parameter size.
  arXiv  Detail & Related papers  (2023-07-28T07:25:57Z)
• Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning [32.69028093984526]
  We propose a novel language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. We achieve within 5% of full finetuning performance on GLUE tasks with as few as 4,100 parameters per task. Our method achieves the best or comparable utility compared to several recent finetuning methods when training with the same privacy constraints.
  arXiv  Detail & Related papers  (2023-05-30T17:55:06Z)
• Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptation [53.835365470800916]
  We show how to efficiently and effectively transfer knowledge in a vision transformer. We propose consolidator to modify the pre-trained model with the addition of a small set of tunable parameters. Our consolidator can reach up to 7.56 better accuracy than full fine-tuning with merely 0.35% parameters.
  arXiv  Detail & Related papers  (2023-04-30T23:59:02Z)
• AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks [55.705355299065474]
  Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed.
  arXiv  Detail & Related papers  (2022-04-30T16:49:41Z)
• Parameter-Efficient Transfer Learning with Diff Pruning [108.03864629388404]
  diff pruning is a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark.
  arXiv  Detail & Related papers  (2020-12-14T12:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.