A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2604.04488v1
• Date: Mon, 06 Apr 2026 07:27:04 GMT
• Title: A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models
• Abstract summary: We propose a unified defense framework based on patch augmentation and cross-view regularity.<n>Our work enables the secure, controlled deployment of large-scale multimodal models in realistic low-frequency poisoning and covert triggering scenarios.
• Score: 10.372037861522783
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models have become an important infrastructure for unified processing of visual and linguistic tasks. However, such models are highly susceptible to backdoor implantation during supervised fine-tuning and will steadily output the attacker's predefined harmful responses once a specific trigger pattern is activated. The core challenge of backdoor defense lies in suppressing attack success under low poisoning ratios while preserving the model's normal generation ability. These two objectives are inherently conflicting. Strong suppression often degrades benign performance, whereas weak regularization fails to mitigate backdoor behaviors. To this end, we propose a unified defense framework based on patch augmentation and cross-view regularity, which simultaneously constrains the model's anomalous behaviors in response to triggered patterns from both the feature representation and output distribution levels. Specifically, patch-level data augmentation is combined with cross-view output difference regularization to exploit the fact that backdoor responses are abnormally invariant to non-semantic perturbations and to proactively pull apart the output distributions of the original and perturbed views, thereby significantly suppressing the success rate of backdoor triggering. At the same time, we avoid over-suppression of the model during defense by imposing output entropy constraints, ensuring the quality of normal command generation. Experimental results across three models, two tasks, and six attacks show that our proposed defense method effectively reduces the attack success rate while maintaining a high level of normal text generation capability. Our work enables the secure, controlled deployment of large-scale multimodal models in realistic low-frequency poisoning and covert triggering scenarios.

Related papers

• BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks [0.0]
  Supervised fine-tuning is the predominant approach for adapting autoregressive vision-language models to downstream tasks.<n>Recent work has shown that this paradigm is highly vulnerable to backdoor attacks.<n>We propose a backdoor-robust fine-tuning framework motivated by the observation that poisoned target responses are often semantically implausible.
  arXiv  Detail & Related papers  (2026-06-01T22:58:04Z)
• Stealthy and Adjustable Text-Guided Backdoor Attacks on Multimodal Pretrained Models [65.63901005568772]
  Multimodal pretrained models are vulnerable to backdoor attacks.<n>We propose a novel Text-Guided Backdoor (TGB) attack on multimodal pretrained models.
  arXiv  Detail & Related papers  (2026-04-07T12:46:23Z)
• When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models [32.77084507646192]
  We investigate the phenomenon of Backdoor Modality Collapse in multimodal diffusion models.<n>Attacks often collapse into subset-modality dominance, and cross-modal interaction is negligible or even negative.<n>These findings suggest that high attack success rates often mask a fundamental reliance on a subset of modalities.
  arXiv  Detail & Related papers  (2026-03-06T17:42:08Z)
• BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Models [10.286339414754499]
  Bad RSSD is the first backdoor attack targeting the representation layer of self-supervised diffusion models.<n>It hijacks the semantic representations of poisoned samples with triggers in PCA space toward those of a target image.<n>Bad RSSD substantially outperforms existing attacks in both FID and MSE metrics.
  arXiv  Detail & Related papers  (2026-03-01T09:56:26Z)
• Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models [74.1970982768771]
  We show that well-established data-poisoning pipelines can successfully implant backdoors into MDLMs.<n>We introduce a backdoor defense framework for MDLMs named DiSP (Diffusion Self-Purification)
  arXiv  Detail & Related papers  (2026-02-24T15:47:52Z)
• Backdoor Unlearning by Linear Task Decomposition [69.91984435094157]
  Foundation models are highly susceptible to adversarial perturbations and targeted backdoor attacks.<n>Existing backdoor removal approaches rely on costly fine-tuning to override the harmful behavior.<n>This raises the question of whether backdoors can be removed without compromising the general capabilities of the models.
  arXiv  Detail & Related papers  (2025-10-16T16:18:07Z)
• Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Models [75.29749026964154]
  Ourmethod reduces the average Attack Success Rate to 4.41% across multiple benchmarks.<n>Clean accuracy and utility are preserved within 0.5% of the original model.<n>The defense generalizes across different types of backdoors, confirming its robustness in practical deployment scenarios.
  arXiv  Detail & Related papers  (2025-10-11T15:47:35Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models [70.03122709795122]
  Backdoor attacks targeting text-to-image diffusion models have advanced rapidly.<n>Current backdoor samples often exhibit two key abnormalities compared to benign samples.<n>We propose Trigger without Trace (TwT) by explicitly mitigating these consistencies.
  arXiv  Detail & Related papers  (2025-03-22T10:41:46Z)
• REFINE: Inversion-Free Backdoor Defense via Model Reprogramming [60.554146386198376]
  Backdoor attacks on deep neural networks (DNNs) have emerged as a significant security threat.<n>We propose REFINE, an inversion-free backdoor defense method based on model reprogramming.
  arXiv  Detail & Related papers  (2025-02-22T07:29:12Z)
• CROW: Eliminating Backdoors from Large Language Models via Internal Consistency Regularization [7.282200564983221]
  Large Language Models (LLMs) are vulnerable to backdoor attacks that manipulate outputs via hidden triggers.<n>We propose Internal Consistency Regularization (CROW), a defense leveraging the observation that backdoored models exhibit unstable layer-wise hidden representations when triggered.<n>CROW enforces consistency across layers via adversarial perturbations and regularization during finetuning, neutralizing backdoors without requiring clean reference models or trigger knowledge--only a small clean dataset.
  arXiv  Detail & Related papers  (2024-11-18T07:52:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.