When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models

• URL: http://arxiv.org/abs/2603.27759v2
• Date: Thu, 02 Apr 2026 07:53:17 GMT
• Title: When Surfaces Lie: Exploiting Wrinkle-Induced Attention Shift to Attack Vision-Language Models
• Abstract summary: We propose a parametric structural perturbation method inspired by the mechanics of three-dimensional fabric wrinkles.<n>Our method generates non-rigid perturbations by constructing multi-scale wrinkle fields and integrating displacement field distortion with surface-consistent appearance variations.
• Score: 10.673331719964184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual-Language Models (VLMs) have demonstrated exceptional cross-modal understanding across various tasks, including zero-shot classification, image captioning, and visual question answering. However, their robustness to physically plausible non-rigid deformations-such as wrinkles on flexible surfaces-remains poorly understood. In this work, we propose a parametric structural perturbation method inspired by the mechanics of three-dimensional fabric wrinkles. Specifically, our method generates photorealistic non-rigid perturbations by constructing multi-scale wrinkle fields and integrating displacement field distortion with surface-consistent appearance variations. To achieve an optimal balance between visual naturalness and adversarial effectiveness, we design a hierarchical fitness function in a low-dimensional parameter space and employ an optimization-based search strategy. We evaluate our approach using a two-stage framework: perturbations are first optimized on a zero-shot classification proxy task and subsequently assessed for transferability on generative tasks. Experimental results demonstrate that our method significantly degrades the performance of various state-of-the-art VLMs, consistently outperforming baselines in both image captioning and visual question-answering tasks.

Related papers

• STABLE: Simulation-Ready Tabletop Layout Generation via a Semantics-Physics Dual System [76.3920413336862]
  We present STABLE, a semantics-physics dual-system tailored for simulation-ready tabletop scene generation.<n>STABLE consists of two complementary modules: a Semantic Reasoner and a Physics Corrector.<n> Experiments demonstrate that STABLE successfully generates simulation-ready tabletop scenes that strictly conform to task instructions.
  arXiv  Detail & Related papers  (2026-05-15T16:18:42Z)
• Robust 4D Visual Geometry Transformer with Uncertainty-Aware Priors [38.939519059502025]
  We present a framework designed to disentangle dynamic and static components by modeling uncertainty across different stages of the reconstruction process.<n>Our approach reduces Mean Accuracy error by 13.43% and improves segmentation F-measure by 10.49%.<n>Our framework maintains the efficiency of feed-forward inference and requires no task-specific fine-tuning or per-scene optimization.
  arXiv  Detail & Related papers  (2026-04-10T14:36:50Z)
• Skeletonization-Based Adversarial Perturbations on Large Vision Language Model's Mathematical Text Recognition [2.729898906885749]
  This work explores the visual capabilities and limitations of foundation models by introducing a novel adversarial attack method.<n>Our approach specifically targets images containing text, particularly mathematical formula images, which are more challenging due to their conversion and intricate structure.
  arXiv  Detail & Related papers  (2026-01-08T09:15:27Z)
• Rectifying Latent Space for Generative Single-Image Reflection Removal [16.341477336909765]
  Single-image removal is a highly ill-posed problem, where existing methods struggle to reason about the composition of corrupted regions.<n>This work reframes an editing-purpose latent diffusion model to effectively perceive and process highly ambiguous, layered image inputs.
  arXiv  Detail & Related papers  (2025-12-06T09:16:14Z)
• Edit2Perceive: Image Editing Diffusion Models Are Strong Dense Perceivers [55.15722080205737]
  Edit2Perceive is a unified diffusion framework that adapts editing models for depth, normal, and matting.<n>Our single-step deterministic inference yields up to faster runtime while training on relatively small datasets.
  arXiv  Detail & Related papers  (2025-11-24T01:13:51Z)
• DCEvo: Discriminative Cross-Dimensional Evolutionary Learning for Infrared and Visible Image Fusion [58.36400052566673]
  Infrared and visible image fusion integrates information from distinct spectral bands to enhance image quality.<n>Existing approaches treat image fusion and subsequent high-level tasks as separate processes.<n>We propose a Discriminative Cross- Dimension Evolutionary Learning Framework, termed DCEvo, which simultaneously enhances visual quality and perception accuracy.
  arXiv  Detail & Related papers  (2025-03-22T07:01:58Z)
• Visual Prompt Tuning in Null Space for Continual Learning [51.96411454304625]
  Existing prompt-tuning methods have demonstrated impressive performances in continual learning (CL) This paper aims to learn each task by tuning the prompts in the direction orthogonal to the subspace spanned by previous tasks' features. In practice, an effective null-space-based approximation solution has been proposed to implement the prompt gradient projection.
  arXiv  Detail & Related papers  (2024-06-09T05:57:40Z)
• What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? [49.84679952948808]
  Recent works show promising results by simply fine-tuning T2I diffusion models for dense perception tasks.<n>We conduct a thorough investigation into critical factors that affect transfer efficiency and performance when using diffusion priors.<n>Our work culminates in the development of GenPercept, an effective deterministic one-step fine-tuning paradigm tailed for dense visual perception tasks.
  arXiv  Detail & Related papers  (2024-03-10T04:23:24Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis [15.76266032768078]
  This paper introduces innovative solutions to enhance spatial controllability in diffusion models reliant on text queries.<n>We first introduce vision guidance as a foundational spatial cue within the perturbed distribution.<n>We propose a universal framework, Layered Rendering Diffusion (LRDiff), which constructs an image-rendering process with multiple layers.
  arXiv  Detail & Related papers  (2023-11-30T10:36:19Z)
• Learning visual-based deformable object rearrangement with local graph neural networks [4.333220038316982]
  We propose a novel representation strategy that can efficiently model the deformable object states with a set of keypoints and their interactions. We also propose a light local GNN learning to jointly model the deformable rearrangement dynamics and infer the optimal manipulation actions. Our method reaches much higher success rates on a variety of deformable rearrangement tasks (96.3% on average) than state-of-the-art method in simulation experiments.
  arXiv  Detail & Related papers  (2023-10-16T11:42:54Z)
• Differentiable Rendering with Perturbed Optimizers [85.66675707599782]
  Reasoning about 3D scenes from their 2D image projections is one of the core problems in computer vision. Our work highlights the link between some well-known differentiable formulations and randomly smoothed renderings. We apply our method to 3D scene reconstruction and demonstrate its advantages on the tasks of 6D pose estimation and 3D mesh reconstruction.
  arXiv  Detail & Related papers  (2021-10-18T08:56:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.