Securing Multimodal AI through Internal Information Decomposition

• URL: http://arxiv.org/abs/2607.21600v1
• Date: Sun, 03 May 2026 19:21:06 GMT
• Title: Securing Multimodal AI through Internal Information Decomposition
• Abstract summary: Multimodal large language models introduce attack surfaces absent in unimodal systems.<n>We propose FlowGuard, a lightweight inference-time framework that detects harmful inputs by monitoring internal multimodal consistency.<n>Our results demonstrate that monitoring cross-modal consistency offers an efficient and effective defense for multimodal reasoning.
• Score: 43.51030662124641
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language models introduce attack surfaces absent in unimodal systems: adversaries can distribute malicious intent across modalities to evade unimodal safeguards. This motivates using cross-modal consistency as a detection signal rather than inspecting each modality in isolation. Our key observation is that benign inputs induce compatible predictive behavior from text-only and vision-only reasoning that stabilizes when fused, whereas adversarial manipulation disrupts this consistency, causing abnormal multimodal behavior. Existing defenses that examine raw inputs or outputs overlook this internal fusion process, rendering them brittle and computationally expensive. We propose FlowGuard, a lightweight inference-time framework that detects harmful inputs by monitoring internal multimodal consistency. Unlike approaches that rely on scalar confidence metrics, FlowGuard derives FlowVectors inspired by Partial Information Decomposition that quantify cross-modal redundancy, synergy, and modality-specific dominance, capturing whether fused multimodal predictions remain aligned with unimodal semantic evidence. In a one-class classification problem trained solely on benign data, FlowGuard reduces Attack Success Rates from >90% to <15% on unseen attacks, with <3% utility loss and up to a 6 times latency reduction. Our results demonstrate that monitoring cross-modal consistency offers an efficient and effective defense for multimodal reasoning.

Related papers

• Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction [43.59114552026716]
  Multimodal large language models (MLLMs) often fail to transfer safety capabilities learned in the text modality to semantically equivalent non-text inputs.<n>We study this gap by analyzing a text-aligned refusal direction and a modality-induced drift direction.<n>We propose ReGap, a training-free inference-time method that adaptively corrects modality drift using self-rectification.
  arXiv  Detail & Related papers  (2026-05-18T09:16:55Z)
• Adversarial Evasion in Non-Stationary Malware Detection: Minimizing Drift Signals through Similarity-Constrained Perturbations [4.8475753151256695]
  Our research investigates a fundamental security question: Can an attacker generate adversarial malware samples that simultaneously evade classification and remain inconspicuous to drift monitoring mechanisms?<n>We propose a novel approach that generates adversarial examples in the classifier's standardized feature space, augmented with sophisticated similarity regularizers.<n>We observe that perturbation budget significantly influences the evasion trade-off, with higher attack success rates and more substantial drift indicators.
  arXiv  Detail & Related papers  (2026-04-23T06:03:50Z)
• When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models [32.77084507646192]
  We investigate the phenomenon of Backdoor Modality Collapse in multimodal diffusion models.<n>Attacks often collapse into subset-modality dominance, and cross-modal interaction is negligible or even negative.<n>These findings suggest that high attack success rates often mask a fundamental reliance on a subset of modalities.
  arXiv  Detail & Related papers  (2026-03-06T17:42:08Z)
• CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs [10.42126976065225]
  Multimodal large language models (MLLMs) enable interaction over both text and images.<n>This paper introduces CSR-Bench, a benchmark for evaluating cross-modal reliability.<n>We evaluate 16 state-of-the-art MLLMs and observe systematic cross-modal alignment gaps.
  arXiv  Detail & Related papers  (2026-02-03T08:49:44Z)
• Contamination Detection for VLMs using Multi-Modal Semantic Perturbation [73.76465227729818]
  Open-source Vision-Language Models (VLMs) have achieved state-of-the-art performance on benchmark tasks.<n>Pretraining corpora raise a critical concern for both practitioners and users: inflated performance due to test-set leakage.<n>We show that existing detection approaches either fail outright or exhibit inconsistent behavior.<n>We propose a novel simple yet effective detection method based on multi-modal semantic perturbation.
  arXiv  Detail & Related papers  (2025-11-05T18:59:52Z)
• DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models [55.30555646945055]
  Text-to-Image (T2I) models are vulnerable to semantic leakage.<n>We introduce DeLeaker, a lightweight approach that mitigates leakage by directly intervening on the model's attention maps.<n>SLIM is the first dataset dedicated to semantic leakage.
  arXiv  Detail & Related papers  (2025-10-16T17:39:21Z)
• CoDefend: Cross-Modal Collaborative Defense via Diffusion Purification and Prompt Optimization [4.6467356929461925]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success in tasks such as image captioning, visual question answering, and cross-modal reasoning.<n>Their multimodal nature exposes them to adversarial threats, where attackers can perturb either modality or both jointly to induce harmful, misleading, or policy violating outputs.<n>Existing defense strategies, such as adversarial training and input purification, face notable limitations.<n>We propose a supervised diffusion based denoising framework that leverages paired adversarial clean image datasets to fine-tune diffusion models.
  arXiv  Detail & Related papers  (2025-10-13T07:44:54Z)
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models [50.21378052667732]
  We conduct an in-depth analysis of dLLM vulnerabilities to jailbreak attacks across two distinct dimensions: intra-step and inter-step dynamics.<n>We propose DiffuGuard, a training-free defense framework that addresses vulnerabilities through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-09-29T05:17:10Z)
• Backdoor Cleaning without External Guidance in MLLM Fine-tuning [76.82121084745785]
  Believe Your Eyes (BYE) is a data filtering framework that leverages attention entropy patterns as self-supervised signals to identify and filter backdoor samples.<n>It achieves near-zero attack success rates while maintaining clean-task performance.
  arXiv  Detail & Related papers  (2025-05-22T17:11:58Z)
• Lie Detector: Unified Backdoor Detection via Cross-Examination Framework [68.45399098884364]
  We propose a unified backdoor detection framework in the semi-honest setting.<n>Our method achieves superior detection performance, improving accuracy by 5.4%, 1.6%, and 11.9% over SoTA baselines.<n> Notably, it is the first to effectively detect backdoors in multimodal large language models.
  arXiv  Detail & Related papers  (2025-03-21T06:12:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.