Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models

• URL: http://arxiv.org/abs/2601.12042v1
• Date: Sat, 17 Jan 2026 13:02:41 GMT
• Title: Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models
• Authors: Xiaomei Zhang, Zhaoxi Zhang, Leo Yu Zhang, Yanjun Zhang, Guanhong Tao, Shirui Pan,
• Abstract summary: We show that visual token compression substantially degrades the robustness of Large Vision-Language Models (LVLMs)<n>Small and imperceptible perturbations can significantly alter token importance ranking, leading the compression mechanism to mistakenly discard task-critical information.<n>We propose a Compression-Aware Attack to systematically study and exploit this vulnerability.
• Score: 69.84867664371826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual token compression is widely adopted to improve the inference efficiency of Large Vision-Language Models (LVLMs), enabling their deployment in latency-sensitive and resource-constrained scenarios. However, existing work has mainly focused on efficiency and performance, while the security implications of visual token compression remain largely unexplored. In this work, we first reveal that visual token compression substantially degrades the robustness of LVLMs: models that are robust under uncompressed inference become highly vulnerable once compression is enabled. These vulnerabilities are state-specific; failure modes emerge only in the compressed setting and completely disappear when compression is disabled, making them particularly hidden and difficult to diagnose. By analyzing the key stages of the compression process, we identify instability in token importance ranking as the primary cause of this robustness degradation. Small and imperceptible perturbations can significantly alter token rankings, leading the compression mechanism to mistakenly discard task-critical information and ultimately causing model failure. Motivated by this observation, we propose a Compression-Aware Attack to systematically study and exploit this vulnerability. CAA directly targets the token selection mechanism and induces failures exclusively under compressed inference. We further extend this approach to more realistic black-box settings and introduce Transfer CAA, where neither the target model nor the compression configuration is accessible. We further evaluate potential defenses and find that they provide only limited protection. Extensive experiments across models, datasets, and compression methods show that visual token compression significantly undermines robustness, revealing a previously overlooked efficiency-security trade-off.

Related papers

• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression [22.436953683970007]
  We show that existing encoder-based attacks can substantially overestimate the robustness of compressed vision-language models (LVLMs)<n>We propose the Compression-AliGnEd attack (CAGE), which aligns perturbation optimization with compression inference without assuming access to the deployed compression mechanism or its token budget.
  arXiv  Detail & Related papers  (2026-01-29T10:47:21Z)
• Embodied Image Compression [105.9462341161654]
  This paper introduces, for the first time, the scientific problem of Embodied Image Compression.<n>We establish a standardized benchmark, EmbodiedComp, to facilitate systematic evaluation under ultra-low conditions in a closed-loop setting.<n>We demonstrate that existing Vision-Language-Action models fail to reliably perform even simple manipulation tasks when compressed below the Embodied threshold.
  arXiv  Detail & Related papers  (2025-12-12T14:49:34Z)
• T-MLA: A Targeted Multiscale Log--Exponential Attack Framework for Neural Image Compression [6.189705043887372]
  We propose a more advanced class of vulnerabilities by introducing T-MLA, the first targeted multiscale log--exponential attack framework.<n>Our approach crafts adversarial perturbations in the wavelet domain by directly targeting the quality of the attacked and reconstructed images.<n>Our findings reveal a critical security flaw at the core of generative and content delivery pipelines.
  arXiv  Detail & Related papers  (2025-11-02T21:06:33Z)
• Joint Lossless Compression and Steganography for Medical Images via Large Language Models [63.454510290574355]
  We propose a novel joint lossless compression and steganography framework for medical images.<n>Inspired by bit plane slicing (BPS), we find it feasible to embed privacy messages into medical images in an invisible manner.
  arXiv  Detail & Related papers  (2025-08-03T14:45:51Z)
• Token-Efficient Prompt Injection Attack: Provoking Cessation in LLM Reasoning via Adaptive Token Compression [12.215295420714787]
  "Reasoning Interruption Attack" is a prompt injection attack based on adaptive token compression.<n>We develop a systematic approach to efficiently collect attack prompts and an adaptive token compression framework.<n> Experiments show our compression framework significantly reduces prompt length while maintaining effective attack capabilities.
  arXiv  Detail & Related papers  (2025-04-29T07:34:22Z)
• Human Aligned Compression for Robust Models [18.95453617434051]
  Adversarial attacks on image models threaten system robustness by introducing imperceptible perturbations that cause incorrect predictions.<n>We investigate human-aligned learned lossy compression as a defense mechanism, comparing two learned models (HiFiC and ELIC) against traditional JPEG across various quality levels.
  arXiv  Detail & Related papers  (2025-04-16T17:05:58Z)
• UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective [85.08718140718707]
  UNComp is an uncertainty-aware framework that uncovers sparsity patterns that can be used for adaptive compression.<n>By focusing on uncertainty to analyze the sparsity pattern in detail, UNComp reduces the KV cache size to 4.74% of the original, achieves a 6% prefill speedup, and improves throughput by 6.4x.
  arXiv  Detail & Related papers  (2024-10-04T02:32:36Z)
• Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression [33.45167213570976]
  We investigate the impact of model compression on four dimensions: (1) degeneration harm, i.e., bias and toxicity in generation; (2) representational harm, i.e., biases in discriminative tasks; (3) dialect bias; and(4) language modeling and downstream task performance. Our analysis reveals that compression can lead to unexpected consequences.
  arXiv  Detail & Related papers  (2024-07-06T05:56:22Z)
• Transferable Learned Image Compression-Resistant Adversarial Perturbations [66.46470251521947]
  Adversarial attacks can readily disrupt the image classification system, revealing the vulnerability of DNN-based recognition tasks. We introduce a new pipeline that targets image classification models that utilize learned image compressors as pre-processing modules.
  arXiv  Detail & Related papers  (2024-01-06T03:03:28Z)
• Robustness and Transferability of Universal Attacks on Compressed Models [3.187381965457262]
  Neural network compression methods like pruning and quantization are very effective at efficiently deploying Deep Neural Networks (DNNs) on edge devices. In particular, Universal Adversarial Perturbations (UAPs), are a powerful class of adversarial attacks. We show that, in some scenarios, quantization can produce gradient-masking, giving a false sense of security.
  arXiv  Detail & Related papers  (2020-12-10T23:40:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.