When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models

• URL: http://arxiv.org/abs/2603.06508v1
• Date: Fri, 06 Mar 2026 17:42:08 GMT
• Title: When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models
• Abstract summary: We investigate the phenomenon of Backdoor Modality Collapse in multimodal diffusion models.<n>Attacks often collapse into subset-modality dominance, and cross-modal interaction is negligible or even negative.<n>These findings suggest that high attack success rates often mask a fundamental reliance on a subset of modalities.
• Score: 32.77084507646192
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While diffusion models have revolutionized visual content generation, their rapid adoption has underscored the critical need to investigate vulnerabilities, e.g., to backdoor attacks. In multimodal diffusion models, it is natural to expect that attacking multiple modalities simultaneously (e.g., text and image) would yield complementary effects and strengthen the overall backdoor. In this paper, we challenge this assumption by investigating the phenomenon of Backdoor Modality Collapse, a scenario where the backdoor mechanism degenerates to rely predominantly on a subset of modalities, rendering others redundant. To rigorously quantify this behavior, we introduce two novel metrics: Trigger Modality Attribution (TMA) and Cross-Trigger Interaction (CTI). Through extensive experiments across diverse training configurations in multimodal conditional diffusion, we consistently observe a ``winner-takes-all'' dynamic in backdoor behavior. Our results reveal that (1) attacks often collapse into subset-modality dominance, and (2) cross-modal interaction is negligible or even negative, contradicting the intuition of synergistic vulnerability. These findings highlight a critical blind spot in current assessments, suggesting that high attack success rates often mask a fundamental reliance on a subset of modalities. This establishes a principled foundation for mechanistic analysis and future defense development.

Related papers

• Securing Multimodal AI through Internal Information Decomposition [43.51030662124641]
  Multimodal large language models introduce attack surfaces absent in unimodal systems.<n>We propose FlowGuard, a lightweight inference-time framework that detects harmful inputs by monitoring internal multimodal consistency.<n>Our results demonstrate that monitoring cross-modal consistency offers an efficient and effective defense for multimodal reasoning.
  arXiv  Detail & Related papers  (2026-05-03T19:21:06Z)
• Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities [52.189426539304065]
  This paper introduces a paradigm shift that redefines multimodal forensics from conventional "feature fusion" to "modality generalization"<n>We propose the first modality-agnostic forgery (MAF) detection framework. By explicitly decoupling modality-specific styles, MAF precisely extracts the essential, cross-modal latent forgery knowledge.
  arXiv  Detail & Related papers  (2026-04-09T03:35:21Z)
• Stealthy and Adjustable Text-Guided Backdoor Attacks on Multimodal Pretrained Models [65.63901005568772]
  Multimodal pretrained models are vulnerable to backdoor attacks.<n>We propose a novel Text-Guided Backdoor (TGB) attack on multimodal pretrained models.
  arXiv  Detail & Related papers  (2026-04-07T12:46:23Z)
• A Patch-based Cross-view Regularized Framework for Backdoor Defense in Multimodal Large Language Models [10.372037861522783]
  We propose a unified defense framework based on patch augmentation and cross-view regularity.<n>Our work enables the secure, controlled deployment of large-scale multimodal models in realistic low-frequency poisoning and covert triggering scenarios.
  arXiv  Detail & Related papers  (2026-04-06T07:27:04Z)
• Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Models [53.41258113970795]
  We investigate whether cross-functionality reciprocity itself constitutes a structural source of vulnerability in Unified Multimodal Models (UMMs)<n>We propose RICE: Reciprocal Interaction-based Cross-functionality Exploitation, a novel attack paradigm that explicitly exploits bidirectional interactions between understanding and generation.
  arXiv  Detail & Related papers  (2026-03-28T16:28:39Z)
• Beyond Passive Aggregation: Active Auditing and Topology-Aware Defense in Decentralized Federated Learning [1.8262547855491456]
  Decentralized Learning (DFL) remains highly vulnerable to backdoor attacks designed to bypass traditional defense metrics.<n>To address this limitation, we shift the defensive paradigm toward a novel active, interventional auditing framework.
  arXiv  Detail & Related papers  (2026-03-19T06:46:55Z)
• Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts [74.47786985522762]
  We identify a critical failure mode termed textual inertia, where models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence.<n>We propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs.<n>Results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation.
  arXiv  Detail & Related papers  (2026-01-07T16:39:34Z)
• Learning Representation and Synergy Invariances: A Povable Framework for Generalized Multimodal Face Anti-Spoofing [85.00865662325954]
  Multimodal Face Anti-Spoofing (FAS) methods, which integrate multiple visual modalities, often suffer even more severe performance degradation when deployed in unseen domains.<n>This is mainly due to two overlooked risks that affect cross-domain multimodal generalization.<n>We propose a provable framework, namely Multimodal Representation and Synergy Invariance Learning (RiSe)
  arXiv  Detail & Related papers  (2025-11-18T05:37:06Z)
• Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers [0.0]
  Large language models (LLMs) aligned for safety often exhibit emergent deceptive behaviors.<n>This paper introduces adversarial activation patching, a novel mechanistic interpretability framework.<n>By sourcing activations from "deceptive" prompts, we simulate vulnerabilities and quantify deception rates.
  arXiv  Detail & Related papers  (2025-07-12T21:29:49Z)
• Rethinking Explainability in the Era of Multimodal AI [9.57008593971486]
  multimodal AI systems have become ubiquitous and achieved remarkable performance across high-stakes applications.<n>Most existing explainability techniques remain unimodal, generating modality-specific feature attributions, concepts, or circuit traces in isolation.<n>This paper argues that such unimodal explanations systematically misrepresent and fail to capture the cross-modal influence that drives multimodal model decisions.
  arXiv  Detail & Related papers  (2025-06-16T03:08:29Z)
• A Closer Look at Multimodal Representation Collapse [12.399005128036746]
  We show that modality collapse happens when noisy features from one modality are entangled, via a shared set of neurons in the fusion head, with predictive features from another.<n>We propose an algorithm that prevents modality collapse through explicit basis reallocation, with applications in dealing with missing modalities.
  arXiv  Detail & Related papers  (2025-05-28T15:31:53Z)
• Trigger without Trace: Towards Stealthy Backdoor Attack on Text-to-Image Diffusion Models [70.03122709795122]
  Backdoor attacks targeting text-to-image diffusion models have advanced rapidly.<n>Current backdoor samples often exhibit two key abnormalities compared to benign samples.<n>We propose Trigger without Trace (TwT) by explicitly mitigating these consistencies.
  arXiv  Detail & Related papers  (2025-03-22T10:41:46Z)
• BadCM: Invisible Backdoor Attack Against Cross-Modal Learning [110.37205323355695]
  We introduce a novel bilateral backdoor to fill in the missing pieces of the puzzle in the cross-modal backdoor. BadCM is the first invisible backdoor method deliberately designed for diverse cross-modal attacks within one unified framework.
  arXiv  Detail & Related papers  (2024-10-03T03:51:53Z)
• Probing the Robustness of Vision-Language Pretrained Models: A Multimodal Adversarial Attack Approach [30.9778838504609]
  Vision-language pretraining with transformers has demonstrated exceptional performance across numerous multimodal tasks. Existing multimodal attack methods have largely overlooked cross-modal interactions between visual and textual modalities. We propose a novel Joint Multimodal Transformer Feature Attack (JMTFA) that concurrently introduces adversarial perturbations in both visual and textual modalities.
  arXiv  Detail & Related papers  (2024-08-24T04:31:37Z)
• Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift [104.76588209308666]
  This paper explores backdoor attacks in LVLM instruction tuning across mismatched training and testing domains.<n>We introduce a new evaluation dimension, backdoor domain generalization, to assess attack robustness.<n>We propose a multimodal attribution backdoor attack (MABA) that injects domain-agnostic triggers into critical areas.
  arXiv  Detail & Related papers  (2024-06-27T02:31:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.