Multimodal Resource-Exhaustion Attacks on Vision-Language Models via Joint Pixel-Prompt Optimization

• URL: http://arxiv.org/abs/2609.05889v1
• Date: Sat, 05 Sep 2026 05:20:53 GMT
• Title: Multimodal Resource-Exhaustion Attacks on Vision-Language Models via Joint Pixel-Prompt Optimization
• Abstract summary: We introduce Joint Pixel-Prompt Optimization (JPPO), the first compound adversarial framework elevating the visible prompt to a first-class adversarial variable alongside image perturbations.<n>Under a restricted joint-input threat model, JPPO performs coupled, stagewise optimization over both the pixel and prompt surfaces.<n>This produces synergistic cost amplification, mechanistically distinct from loop-dependent failures.
• Score: 38.37373243718745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Resource-exhaustion attacks against autoregressive vision-language models (VLMs) typically assume unimodal threat models, treating the image branch as the primary optimization surface while holding user-visible prompts fixed. Even recent loop-centric variants remain confined to this single-channel paradigm, leaving the exploitation of availability unexplored as a cross-modal optimization problem over jointly controllable input surfaces. We introduce Joint Pixel-Prompt Optimization (JPPO), the first compound adversarial framework elevating the visible prompt to a first-class adversarial variable alongside image perturbations. Under a restricted joint-input threat model, JPPO performs coupled, stagewise optimization over both the pixel and prompt surfaces. This produces synergistic cost amplification, mechanistically distinct from loop-dependent failures, exhibiting negligible loop incidence in our experiments. Evaluating five open-source VLM families on MS COCO and ImageNet under an 8/255 infinity-norm budget, JPPO achieves over 4.6x latency and 5.3x energy amplification on Qwen2.5-VL-7B, and over 36.6x latency with 32.7x energy amplification on BLIP-2. This represents the strongest cost amplification among directly compared baselines while requiring substantially fewer optimization iterations. Ablations confirm this amplification arises from multimodal coordination rather than prompt length or isolated modalities. These findings reveal structural blind spots in current VLM serving defenses, motivating cost-aware robustness evaluation as a first-class security requirement for multimodal deployments.

Related papers

• Structured Redundancy Modeling for Efficient Visual Token Pruning in High-Resolution MLLMs [9.080169376642433]
  We propose a structural reformulation of visual token pruning that embeds redundancy control directly into the scoring space.<n>We show that our approach maintains stable selection costs, reducing the token selection process by up to 110 ms, from 112.4 ms to just 2.5 ms at 512 tokens in Qwen2.5-VL.
  arXiv  Detail & Related papers  (2026-07-25T05:08:22Z)
• Kwai Keye-VL-2.0 Technical Report [53.82434681649277]
  Keye-VL-2.0 is a multimodal foundation model designed to advance long-video understanding and agentic intelligence.<n>It is the first to adapt DeepSeek Sparse Attention (DSA) to GQA-based multimodal architectures.<n>Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) paired with Context-RL and Video-RL overcomes the algorithmic dilemma of catastrophic forgetting.
  arXiv  Detail & Related papers  (2026-06-09T09:58:08Z)
• Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models [0.0]
  We show that first-order preconditioning, particularly SOAP, provides a more stable basis for multimodal alignment.<n>We propose emphML-FOP-SOAP, a second-order optimization framework with Multi-Level Variance Correction.<n>Compared with AdamW, our method improves sample efficiency by up to $1.4times$ and accelerates wall-clock training by up to $1.5times$.
  arXiv  Detail & Related papers  (2026-05-15T16:45:56Z)
• A Cross-Modal Prompt Injection Attack against Large Vision-Language Models with Image-Only Perturbation [14.191447589526796]
  Large vision-language models (LVLMs) have emerged as a powerful paradigm for multimodal intelligence.<n>We introduce a novel cross-modal prompt injection attack CrossMPI.<n>Our design is underpinned by the following key breakthroughs.
  arXiv  Detail & Related papers  (2026-05-15T15:47:41Z)
• Large Language Models as Amortized Pareto-Front Generators for Constrained Bi-Objective Convex Optimization [21.83927813316836]
  DIPS is an end-to-end framework that fine-tunes large language models as amortized generators for constrained bi-objective convex optimization.<n>To make continuous optimization compatible with autoregressive language modeling, DIPS combines a compact discretization scheme, Numerically Grounded Token Initialization for new numerical tokens, and Three-Phase Curriculum Optimization.<n>With vLLM-accelerated inference, DIPS solves one instance in as little as 0.16 seconds and outperforms general-purpose and reasoning LLM baselines.
  arXiv  Detail & Related papers  (2026-05-12T13:20:57Z)
• Hybrid Latent Reasoning with Decoupled Policy Optimization [19.348125016748018]
  HyLaR (Hybrid Latent Reasoning) is a framework that seamlessly interleaves discrete text generation with continuous visual latent representations.<n>We show that HyLaR outperforms state-of-the-art latent reasoning approaches across fine-grained perception and general multimodal understanding benchmarks.
  arXiv  Detail & Related papers  (2026-04-22T08:22:23Z)
• $D^3$-RSMDE: 40$\times$ Faster and High-Fidelity Remote Sensing Monocular Depth Estimation [72.9912717963138]
  Real-time, high-fidelity monocular depth estimation from remote sensing imagery is crucial for numerous applications.<n>Although using Vision Transformer (ViT) backbones for dense prediction is fast, they often exhibit poor perceptual quality.<n>We propose Depth Detail Diffusion for Remote Sensing Monocular Depth Estimation ($D3$-RSMDE) to overcome these limitations.<n>$D3$-RSMDE achieves a notable 11.85% reduction in the Learned Perceptual Image Patch Similarity (LPIPS) perceptual metric.
  arXiv  Detail & Related papers  (2026-03-17T10:50:36Z)
• Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models [67.45032003041399]
  We propose a novel Multi-Paradigm Collaborative Attack (MPCAttack) framework to boost the transferability of adversarial examples against MLLMs.<n>MPCO adaptively balances the importance of different paradigm representations and guides the global optimisation.<n>Our solution consistently outperforms state-of-the-art methods in both targeted and untargeted attacks on open-source and closed-source MLLMs.
  arXiv  Detail & Related papers  (2026-03-05T06:01:26Z)
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment [61.80228667422234]
  VGPO redefines value estimation across both temporal and group dimensions.<n>It transforms the sparse terminal reward into dense, process-aware value estimates.<n>It replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal.
  arXiv  Detail & Related papers  (2025-12-13T16:31:26Z)
• Repulsor: Accelerating Generative Modeling with a Contrastive Memory Bank [65.00301565190824]
  mname is a plug-and-play training framework that requires no external encoders.<n>mname achieves a state-of-the-art FID of textbf2.40 within 400k steps, significantly outperforming comparable methods.
  arXiv  Detail & Related papers  (2025-12-09T14:39:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.