DiffPrune: differentiable information throttling for token pruning in vision-language models

• URL: http://arxiv.org/abs/2608.01985v1
• Date: Mon, 03 Aug 2026 09:46:32 GMT
• Title: DiffPrune: differentiable information throttling for token pruning in vision-language models
• Abstract summary: We propose DiffPrune, which gives token scores a direct meaning.<n>Across ten VLM benchmarks, DiffPrune retains 96.5% of full-model accuracy.
• Score: 8.48481119028537
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Visual token pruning reduces the computational cost of Vision-Language Models (VLMs) by removing redundant visual tokens. The key is to learn a score that measures whether a token is useful. Existing methods typically rely on Gumbel-Softmax to approximate discrete selection during training. Such selectors make the score depend on the behavior of a relaxed pruning operator, not directly on the consequence of information loss. In this paper, we propose DiffPrune, which gives token scores a direct meaning. During training, DiffPrune keeps all tokens and weakens each token's information according to its score. If weakening a token hurts the task, the scorer is pushed to protect it; if not, the token can receive a lower score. Because the loss is differentiated through this actual information-throttling path, the scorer avoids the unstable surrogate path of relaxed token selection. DiffPrune implements this idea with an Information Throttler, which injects variance-preserving noise into visual tokens, where high-score tokens remain close to their original representations, while low-score tokens carry less original information. At inference, the throttler is removed, and hard top-K pruning is applied using the learned scores. Across ten VLM benchmarks, DiffPrune retains 96.5% of full-model accuracy while accelerating LLM prefill by 2.85x, with only 0.69 ms inference overhead. Code will be publicly available.

Related papers

• Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models [8.48481119028537]
  We propose DiffPrune, which reformulates pruning as continuous control of token information instead of discrete selection learning.<n>Across ten VLM benchmarks, DiffPrune retains 96.5% of full-model accuracy while accelerating LLM prefill by 2.85x, with only 0.69 ms of inference overhead.
  arXiv  Detail & Related papers  (2026-05-27T06:52:08Z)
• Localization then Neutralization: Gradient-guided Token Suppression against Visual Prompt Injection Attack [78.02110947708535]
  Adversarial images pose a severe security threat to multimodal large language models through prompt injection.<n>We show that successful adversarial attacks do not rely on the entire image uniformly but instead depend on a small subset of critical image tokens.<n>We propose Gradient Token Masking (GTM), which localizes these tokens via gradient analysis and neutralizes them through masking.
  arXiv  Detail & Related papers  (2026-05-24T17:51:34Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• ID-Selection: Importance-Diversity Based Visual Token Selection for Efficient LVLM Inference [58.28899599773308]
  ID-Selection is a simple yet effective token selection strategy for efficient LVLM inference.<n>It preserves informative tokens while reducing redundancy in a unified selection process.<n>Experiments across 5 LVLM backbones and 16 main benchmarks demonstrate that ID-Selection consistently achieves superior performance and efficiency.
  arXiv  Detail & Related papers  (2026-04-07T08:52:28Z)
• The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating [2.503823246244002]
  Visual tokens dominate inference cost in vision-language models (VLMs)<n>We reformulate visual token pruning as capacity constrained communication.<n>We propose AutoSelect, which attaches a lightweight Scorer and Denoiser to a frozen VLM and trains with only the standard next token prediction loss.
  arXiv  Detail & Related papers  (2026-03-07T09:57:35Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• TokenButler: Token Importance is Predictable [8.514853311344458]
  Large Language Models (LLMs) rely on the Key-Value (KV) Cache to store token history, enabling efficient decoding of tokens.<n>Prior research has shown that only a small subset of tokens contribute meaningfully to each decoding step.<n>We introduce TokenButler, a high-granularity, query-aware predictor that learns to identify these critical tokens.
  arXiv  Detail & Related papers  (2025-03-10T16:41:14Z)
• Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More [18.928285521147057]
  We show that importance is not an ideal indicator to decide whether a token should be pruned.<n>We propose DART (Duplication-Aware Reduction of Tokens), which prunes tokens based on its duplication with other tokens.<n>Experiments demonstrate that DART can prune 88.9% vision tokens while maintaining comparable performance.
  arXiv  Detail & Related papers  (2025-02-17T06:56:28Z)
• Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation [89.88214896713846]
  STA score considers two critical factors: temporal redundancy and semantic importance. We apply the STA module to off-the-shelf video Transformers and Videowins. Results: Kinetics-400 and Something-Something V2 achieve 30% overshelf reduction with a negligible 0.2% accuracy drop.
  arXiv  Detail & Related papers  (2023-08-08T19:38:15Z)
• How can objects help action recognition? [74.29564964727813]
  We investigate how we can use knowledge of objects to design better video models. First, we propose an object-guided token sampling strategy that enables us to retain a small fraction of the input tokens. Second, we propose an object-aware attention module that enriches our feature representation with object information.
  arXiv  Detail & Related papers  (2023-06-20T17:56:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.