From Parameters to Feature Space: Task Arithmetic for Backdoor Mitigation in Model Merging

• URL: http://arxiv.org/abs/2606.12498v1
• Date: Wed, 10 Jun 2026 13:51:36 GMT
• Title: From Parameters to Feature Space: Task Arithmetic for Backdoor Mitigation in Model Merging
• Abstract summary: Model merging (MM) has gained significant attention as a cost-effective approach to integrate multiple task-specific models into a unified model.<n>Recent work reveals that MM is highly susceptible to backdoor attacks.<n>We propose Linear Feature Path Minimization (LFPM), a backdoor mitigation framework for model merging.
• Score: 9.772291868457812
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model merging (MM) has gained significant attention as a cost-effective approach to integrate multiple task-specific models into a unified model. However, recent work reveals that MM is highly susceptible to backdoor attacks. Existing defenses based on task arithmetic often fail to eliminate backdoors without substantially degrading clean-task performance, owing to their reliance on direct parameter-space editing. To address this gap, we propose Linear Feature Path Minimization (LFPM), a backdoor mitigation framework for model merging, which introduces an anti-backdoor task vector into the backdoored merged model. Unlike prior approaches, LFPM formulates the backdoor robustness of the merged model from a unified feature-space perspective under the Cross-Task Linearity (CTL) framework, which leverages the approximate linearity of features across tasks. This perspective guides the optimization of the anti-backdoor task to suppress backdoors while preserving clean-task performance. Furthermore, we introduce an effective optimization mechanism based on gradient accumulation and loss path-integral, ensuring robust backdoor suppression along the interpolation path. Extensive experiments demonstrate that LFPM consistently exhibits strong robustness against backdoor attacks in both full fine-tuning and Parameter-Efficient Fine-Tuning (PEFT) settings.

Related papers

• Not All Tokens Are Equal: Region-Aware Consistency Repair of Backdoors in MLLMs [19.117078494342714]
  Existing model-level backdoor removal methods show limited effectiveness on MLLMs.<n>We present RACER, a model-level repair framework motivated by a key observation: backdoors induce abnormal layer-to-layer evolution in internal representations.<n> RACER decomposes the fused representation into visual and textual token regions, normalizes their layer-wise inconsistency separately, and recomposes them using modality-aware weights over a deep-layer window.
  arXiv  Detail & Related papers  (2026-08-25T10:10:01Z)
• Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models [74.1970982768771]
  We show that well-established data-poisoning pipelines can successfully implant backdoors into MDLMs.<n>We introduce a backdoor defense framework for MDLMs named DiSP (Diffusion Self-Purification)
  arXiv  Detail & Related papers  (2026-02-24T15:47:52Z)
• Model Merging in the Essential Subspace [78.5390284258307]
  Model merging aims to integrate multiple task-specific fine-tuned models into a single multi-task model without additional training.<n>Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models.<n>We propose ESM (Essential Subspace Merging), a robust framework for effective model merging.
  arXiv  Detail & Related papers  (2026-02-23T00:33:38Z)
• Backdoor Unlearning by Linear Task Decomposition [69.91984435094157]
  Foundation models are highly susceptible to adversarial perturbations and targeted backdoor attacks.<n>Existing backdoor removal approaches rely on costly fine-tuning to override the harmful behavior.<n>This raises the question of whether backdoors can be removed without compromising the general capabilities of the models.
  arXiv  Detail & Related papers  (2025-10-16T16:18:07Z)
• MARS: A Malignity-Aware Backdoor Defense in Federated Learning [51.77354308287098]
  Recently proposed state-of-the-art (SOTA) attack, 3DFed, uses an indicator mechanism to determine whether backdoor models have been accepted by the defender.<n>We propose a Malignity-Aware backdooR defenSe (MARS) that leverages backdoor energy to indicate the malicious extent of each neuron.<n>Experiments demonstrate that MARS can defend against SOTA backdoor attacks and significantly outperforms existing defenses.
  arXiv  Detail & Related papers  (2025-09-21T14:50:02Z)
• Backdoor Mitigation via Invertible Pruning Masks [10.393154496941527]
  We propose a novel pruning approach featuring a learned emphselection mechanism to identify parameters critical to both main and backdoor tasks.<n>We formulate this as a bi-level optimization problem that jointly learns selection variables, a sparse invertible mask, and sample-specific backdoor perturbations.<n>Our approach outperforms existing pruning-based backdoor mitigation approaches, maintains strong performance under limited data conditions, and achieves competitive results compared to state-of-the-art fine-tuning approaches.
  arXiv  Detail & Related papers  (2025-09-19T00:32:19Z)
• Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning [57.514786046966265]
  We propose textbfPerturb-and-Merge (P&M), a novel continual learning framework that integrates model merging into the CL paradigm to mitigate forgetting.<n>Our proposed approach achieves state-of-the-art performance on several continual learning benchmark datasets.
  arXiv  Detail & Related papers  (2025-05-28T14:14:19Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• REFINE: Inversion-Free Backdoor Defense via Model Reprogramming [60.554146386198376]
  Backdoor attacks on deep neural networks (DNNs) have emerged as a significant security threat.<n>We propose REFINE, an inversion-free backdoor defense method based on model reprogramming.
  arXiv  Detail & Related papers  (2025-02-22T07:29:12Z)
• Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace [15.457992715866995]
  We propose a novel Defense-Aware Merging (DAM) approach that simultaneously mitigates task interference and backdoor vulnerabilities.<n>Compared to existing merging methods, DAM achieves a more favorable balance between performance and security, reducing the attack success rate by 2-10 percentage points.
  arXiv  Detail & Related papers  (2024-10-17T00:13:31Z)
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors [10.136109501389168]
  LMSanitator is a novel approach for detecting and removing task-agnostic backdoors on Transformer models. LMSanitator achieves 92.8% backdoor detection accuracy on 960 models and decreases the attack success rate to less than 1% in most scenarios.
  arXiv  Detail & Related papers  (2023-08-26T15:21:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.