Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression

• URL: http://arxiv.org/abs/2604.28109v1
• Date: Thu, 30 Apr 2026 16:58:05 GMT
• Title: Auto-FlexSwitch: Efficient Dynamic Model Merging via Learnable Task Vector Compression
• Abstract summary: We propose T-Switch, which decomposes task vectors into three compact components.<n>We then introduce Auto-Switch, a training-free merging scheme that automatically composes task vectors.<n>We present Auto-FlexSwitch, a dynamic model merging approach that supports highly efficient task vector compression.
• Score: 54.12624130407679
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Model merging has attracted attention as an effective path toward multi-task adaptation by integrating knowledge from multiple task-specific models. Among existing approaches, dynamic merging mitigates performance degradation caused by conflicting parameter updates across tasks by flexibly combining task-specific parameters at inference time, thereby maintaining high performance. However, these methods require storing independent parameters for each task, resulting in prohibitive storage overhead. To address this issue, we first experimentally demonstrate that the fine-tuned weight increments (referred to as task vectors) exhibit an impulse-like activation pattern and high robustness to low-bit representations. Driven by this insight, we propose T-Switch, which decomposes task vectors into three compact components: a binary sparse mask, a sign vector, and a scalar scaling factor, achieving high-fidelity approximation at high compression ratios. We then introduce Auto-Switch, a training-free merging scheme that automatically composes task vectors via feature similarity retrieval. Building on this, we develop Auto-Switch, a training-free merging scheme that automatically assembles task vectors through feature similarity retrieval. Furthermore, to transform task vector sparsification and quantization from static rules to adaptive learning, we propose FlexSwitch, a learnable framework which jointly optimizes the compression strategy for each model unit via Learnable Gating Sparsification (LGS) and Bit-width Adaptive Selection (BAS), while employing the Sparsity-Aware Storage Strategy (SASS) to select the optimal storage encoding structure. Finally, by incorporating a K-Nearest Neighbor (KNN) inference scheme with a learnable low-rank metric, we present Auto-FlexSwitch, a dynamic model merging approach that supports highly efficient task vector compression.

Related papers

• One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning [45.405420567098155]
  We present textbfFACET: task-conditioned textbfFetextbfAture transformation with textbfConditiontextbfEd feature consistextbfTency.<n>Our method achieves superior performance while using significantly fewer trainable parameters and GFLOPs.
  arXiv  Detail & Related papers  (2026-08-31T17:03:45Z)
• CrossMambaTuning: Synergistic Spatial and Cross-Layer Adaptation for Machine Vision Compression [4.294434416475728]
  We propose a novel framework named CrossMambaTuning, which integrates State Space Models with cross-layer interaction mechanisms for parameter-efficient fine-tuning.<n>Specifically, we design an efficient Mamba adapter equipped with task-specific prompts and multi-scale branching to precisely capture both local features and global dependencies.<n>Experiments demonstrate that CrossMambaTuning achieves state-of-the-art (SOTA) performance on multiple machine vision tasks, reducing parameter overhead by 72% compared to SOTA methods.
  arXiv  Detail & Related papers  (2026-08-26T09:20:57Z)
• PACT: Preserving Anchored Cores in Task-vectors for Model Merging [68.52455853496585]
  Model merging aims to combine multiple task-specific fine-tuned models into a single multi-task model.<n>Most existing model merging approaches follow the Task Arithmetic paradigm.<n>We propose PACT, which preserves the anchored task-specific cores (i.e., LBW dimensions) within task vectors by aligning their complements with the subspace of the pre-trained weights.
  arXiv  Detail & Related papers  (2026-06-17T02:48:35Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• Rethinking Autoregressive Models for Lossless Image Compression via Hierarchical Parallelism and Progressive Adaptation [75.58269386927076]
  Autoregressive (AR) models are often dismissed as impractical due to prohibitive computational cost.<n>This work re-thinks this paradigm, introducing a framework built on hierarchical parallelism and progressive adaptation.<n> Experiments on diverse datasets (natural, satellite, medical) validate that our method achieves new state-of-the-art compression.
  arXiv  Detail & Related papers  (2025-11-14T06:27:58Z)
• Controllable-LPMoE: Adapting to Challenging Object Segmentation via Dynamic Local Priors from Mixture-of-Experts [16.21786310193235]
  We propose a novel dynamic priors-based fine-tuning paradigm with fewer trainable parameters, dubbed Controllable-LPMoE.<n>We construct a lightweight dynamic mixed local priors extractor that captures diverse local priors from input images through heterogeneous convolutions.<n>We also design a bi-directional interaction adapter that employs cosine-aligned deformable attention and channel-oriented adaptive scale enhancement.
  arXiv  Detail & Related papers  (2025-10-24T03:03:59Z)
• Purifying Task Vectors in Knowledge-Aware Subspace for Model Merging [83.5273168208788]
  Model merging aims to integrate task-specific abilities from individually fine-tuned models into a single model without extra training.<n>The merged model often suffers from notable performance degradation due to the conflicts caused by task-irrelevant redundancy in task vectors.<n>We propose Purifying TAsk Vectors (PAVE) in knowledge-aware subspace to overcome these challenges.
  arXiv  Detail & Related papers  (2025-10-16T14:02:57Z)
• Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution [88.20464308588889]
  We propose a Structural Similarity-Inspired Unfolding (SSIU) method for efficient image SR.<n>This method is designed through unfolding an SR optimization function constrained by structural similarity.<n>Our model outperforms current state-of-the-art models, boasting lower parameter counts and reduced memory consumption.
  arXiv  Detail & Related papers  (2025-06-13T14:29:40Z)
• SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation [28.417947631789783]
  SyMerge is a lightweight framework that jointly optimize one task-specific layer and merging coefficients.<n>SyMerge achieves state-of-the-art results across vision, dense prediction, and NLP benchmarks.
  arXiv  Detail & Related papers  (2024-12-26T07:42:06Z)
• Less is More: Efficient Model Merging with Binary Task Switch [19.622432243113852]
  We introduce Task Switch, which decomposes task vectors into three components.<n>By storing task vectors in a binarized form, T-Switch alleviates parameter conflicts while ensuring efficient task parameter storage.<n>Experiments indicate that our methods achieve significant performance improvements over existing baselines, requiring only 1-3% of the storage space of full-precision parameters.
  arXiv  Detail & Related papers  (2024-11-24T14:47:13Z)
• DeGMix: Efficient Multi-Task Dense Prediction with Deformable and Gating Mixer [129.61363098633782]
  We present an efficient multi-task dense prediction with deformable and gating mixer (DeGMix)<n>The proposed DeGMix uses fewer GFLOPs and significantly outperforms current Transformer-based and CNN-based competitive models.
  arXiv  Detail & Related papers  (2023-08-10T17:37:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.