On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

• URL: http://arxiv.org/abs/2607.07375v1
• Date: Wed, 08 Jul 2026 13:06:20 GMT
• Title: On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces
• Abstract summary: Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems.<n>Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability.<n>We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors.
• Score: 0.7419725234099728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial vulnerability in deep neural networks (DNNs) has been studied from the perspectives of decision-boundary geometry, feature robustness, input-output Jacobians, and the instability of inverse problems. Here, we focus on the spectral structure of intermediate linear transformations that propagate information through modern DNNs, an unexplored mechanism of adversarial vulnerability. Specifically, we investigate transformer-based vision-language models, whose linear layers admit interpretable spectral decompositions and whose widespread adoption makes understanding their robustness increasingly important. We propose a white-box spectral-subspace-guided attack (SSGRA) that aligns intermediate representations with the subspace spanned by the bottom right singular vectors. Our experiments show improved attack effectiveness over existing baselines. In addition, SSGRA offers a spectral interpretation of adversarial vulnerability in VLMs, providing insights for improving their robustness.

Related papers

• Forbid Your Attention: Fooling Multimodal Large Language Models by Selectively Removing Intrinsic Focus in Spectral Domain [94.64605746540472]
  Multimodal large language models (MLLMs) have extended the capability of large language models (LLMs) to process more contextual multimodal information.<n>Recent studies reveal that MLLMs remain highly vulnerable to adversarial inputs, especially those targeting visual components.<n>We propose a novel phase-aware adversarial attack framework that explicitly restricts adversarial perturbations to structure-relevant phase regions.
  arXiv  Detail & Related papers  (2026-09-01T06:34:32Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• MirrorLA: Reflecting Feature Map for Vision Linear Attention [49.41670925034762]
  Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance.<n>We propose MirrorLA, a geometric framework that substitutes passive truncation with active reorientation.<n>MirrorLA achieves state-of-the-art performance across standard benchmarks, demonstrating that strictly linear efficiency can be achieved without compromising representational fidelity.
  arXiv  Detail & Related papers  (2026-02-04T09:14:09Z)
• FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction [82.6826848085638]
  Visual jailbreaking attacks can manipulate open-source MLLMs more readily than sophisticated textual attacks.<n>These attacks exhibit extremely limited cross-model transferability, failing to reliably identify vulnerabilities in closed-source MLLMs.<n>We propose a Feature Over-Reliance CorrEction (FORCE) method, which guides the attack to explore broader feasible regions.
  arXiv  Detail & Related papers  (2025-09-25T11:36:56Z)
• Rethinking Spatio-Temporal Anomaly Detection: A Vision for Causality-Driven Cybersecurity [22.491097360752903]
  We advocate for a causal learning perspective to advance anomaly detection in spatially distributed infrastructures.<n>We identify and formalize three key directions: causal graph profiling, multi-view fusion, and continual causal graph learning.<n>Our objective is to lay a new research trajectory toward scalable, adaptive, explainable, and spatially grounded anomaly detection systems.
  arXiv  Detail & Related papers  (2025-07-10T21:19:28Z)
• Structure before the Machine: Input Space is the Prerequisite for Concepts [11.573510865918863]
  High-level representations have become a central focus in enhancing AI transparency and control, shifting attention from individual neurons or circuits to structured semantic directions that align with human-interpretable concepts.<n>Motivated by the Linear Representation Hypothesis (LRH), we propose the Input-Space Linearity Hypothesis (ISLH), which posits that concept-aligned directions originate in the input space and are selectively amplified with increasing depth.<n>We then introduce the Spectral Principal Path (SPP) framework, which formalizes how deep networks progressively distill linear representations along a small set of dominant spectral directions.
  arXiv  Detail & Related papers  (2025-06-10T08:08:52Z)
• The Shape of Adversarial Influence: Characterizing LLM Latent Spaces with Persistent Homology [4.280045926995889]
  This study focuses on how adversarial inputs systematically affect the internal representation spaces of Large Language Models.<n>By quantifying the shape of activations and neuronal information flow, our architecture-agnostic framework reveals fundamental invariants of representational change.
  arXiv  Detail & Related papers  (2025-05-26T18:31:49Z)
• Hallucination Detection in LLMs with Topological Divergence on Attention Graphs [60.83579255387347]
  Hallucination, i.e., generating factually incorrect content, remains a critical challenge for large language models.<n>We introduce TOHA, a TOpology-based HAllucination detector in the RAG setting.
  arXiv  Detail & Related papers  (2025-04-14T10:06:27Z)
• Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals [52.123343364599094]
  adversarial attacks place carefully crafted perturbations on normal examples to fool deep neural networks (DNNs) We first empirically show that the features of either clean signals or adversarial perturbations are redundant and span in low-dimensional linear subspaces respectively with minimal overlap. This makes it possible for DNNs to learn a subspace where only features of clean signals exist while those of perturbations are discarded.
  arXiv  Detail & Related papers  (2024-03-24T14:35:44Z)
• Occlusion Sensitivity Analysis with Augmentation Subspace Perturbation in Deep Feature Space [7.021872917042116]
  We introduce the Occlusion Sensitivity Analysis with Deep Feature Augmentation Subspace (OSA-DAS), a novel perturbation-based interpretability approach for computer vision. Our method utilizes the output vector of a DNN to build low-dimensional subspaces within the deep feature vector space, offering a more precise explanation of the model prediction. We test extensively on the ImageNet-1k, and our class- and model-agnostic approach outperforms commonly used interpreters.
  arXiv  Detail & Related papers  (2023-11-25T13:26:40Z)
• Visual Adversarial Examples Jailbreak Aligned Large Language Models [66.53468356460365]
  We show that the continuous and high-dimensional nature of the visual input makes it a weak link against adversarial attacks. We exploit visual adversarial examples to circumvent the safety guardrail of aligned LLMs with integrated vision. Our study underscores the escalating adversarial risks associated with the pursuit of multimodality.
  arXiv  Detail & Related papers  (2023-06-22T22:13:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.