DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservation

• URL: http://arxiv.org/abs/2606.16633v1
• Date: Mon, 15 Jun 2026 12:24:07 GMT
• Title: DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservation
• Abstract summary: Recent vision token pruning methods effectively preserve model performance under moderate token budgets but become unstable under ultra-low token budget.<n>We propose a two-stage pruning framework consisting of Anchor-Context Graph Recovery (ACGR) and Text-Aware Token Cluster Selection (TATCS)<n>Experiments demonstrate that our method achieves superior and more stable performance under ultra-low token budget.
• Score: 55.38884985761757
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent vision token pruning methods effectively preserve model performance under moderate token budgets but become unstable under ultra-low token budget. Our analysis shows that as the pruning budget decreases, accuracy degradation is often accompanied by larger feature distribution shifts. Critically, the degree of this distribution shift strongly correlates with performance degradation. To better characterize this phenomenon, we introduce a lightweight distribution consistency metric to estimate the distribution shift between retained and full tokens. Motivated by these observations, we propose a two-stage pruning framework consisting of Anchor-Context Graph Recovery (ACGR) and Text-Aware Token Cluster Selection (TATCS). Specifically, ACGR transfers contextual information before token removal, while TATCS dynamically re-selects representative tokens when severe distribution shift is detected. Extensive experiments demonstrate that our method achieves superior and more stable performance under ultra-low token budget. Notably, it retains 92.1% of the upper-bound average performance on LLaVA-1.5-7B with only 16 visual tokens.

Related papers

• Analytic Distribution of Classifier-Free Guidance for Schedule Design [3.2049543530625098]
  We analyze CFG through the flow ODE and derive exact analytic path-integral representations of the induced distributions for both constant and time-dependent guidance.<n>This characterization explains how score discrepancies accumulate along sampling trajectories and motivates Distribution-GuidedG (DG-CFG)<n>On Stable Diffusion1.5, DG-CFG improves generation and yields a stronger diversity--fidelity trade-off across guidance strengths.
  arXiv  Detail & Related papers  (2026-07-22T03:53:07Z)
• Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation [50.139984798361375]
  We propose a training-free framework that removes redundancy via latent discrepancy.<n> LD-Pruning substantially reduces inference while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
  arXiv  Detail & Related papers  (2026-05-29T19:34:39Z)
• Towards Joint Quantization and Token Pruning of Vision-Language Models [53.978753457744055]
  Token pruning and low-bit quantization are complementary for reducing inference costs.<n>We present a collaborative quantization-and-pruning framework that unifies low-bit inference and deterministic visual-token pruning.<n> Experiments on standard VLM benchmarks show improved robustness over stage-wise baselines under the same low-bit regime.
  arXiv  Detail & Related papers  (2026-04-19T08:18:29Z)
• Beyond Attention Magnitude: Leveraging Inter-layer Rank Consistency for Efficient Vision-Language-Action Models [75.27089289058613]
  We introduce textbfTIES (textbfTau-guided textbfInter-layer textbfEfficient textbfSelection), a dynamic framework guided by inter-layer token ranking consistency.<n>On the CogACT + SIMPLER benchmark, TIES improves average success rates by 6% while reducing token usage by 78%, and demonstrate strong generalization across diverse decoders and benchmarks.
  arXiv  Detail & Related papers  (2026-03-26T02:13:03Z)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity [54.95089105944234]
  Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens.<n>We propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline.<n>We demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT.
  arXiv  Detail & Related papers  (2026-03-10T10:31:58Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models [16.540220733551823]
  Large Vision-Language Models (VLMs) enable strong multimodal reasoning but incur heavy inference costs from redundant visual tokens.<n> Attention-based methods rely on raw attention scores, which are often unstable across layers and heads.<n>We propose ours, a training-free framework built on a simple intuition.
  arXiv  Detail & Related papers  (2025-09-29T14:20:05Z)
• Additive Distributionally Robust Ranking and Selection [0.8283940114367679]
  We propose a simple additive allocation (AA) procedure that aims to exclusively sample the $k + m - 1$ previously hypothesized critical scenarios.<n>We prove that AA is consistent and, surprisingly, achieves additivity in the strongest sense.<n>Results offer new and counterintuitive insights into the additive structure of DRR&S.
  arXiv  Detail & Related papers  (2025-09-07T17:36:29Z)
• Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study [44.170933007736984]
  Vision-Language Models (VLMs) are powerful yet computationally intensive for widespread practical deployments.<n>Current acceleration evaluations primarily target minimal overall performance degradation, overlooking a crucial question: does the accelerated model still give the same answers to the same questions as it did before acceleration?<n>This is vital for stability-centered industrial applications where consistently correct answers for specific, known situations are paramount, such as in AI-based disease diagnosis.
  arXiv  Detail & Related papers  (2025-03-09T22:16:48Z)
• TPC-ViT: Token Propagation Controller for Efficient Vision Transformer [6.341420717393898]
  Vision transformers (ViTs) have achieved promising results on a variety of Computer Vision tasks. Previous approaches that employ gradual token reduction to address this challenge assume that token redundancy in one layer implies redundancy in all the following layers. We propose a novel token propagation controller (TPC) that incorporates two different token-distributions.
  arXiv  Detail & Related papers  (2024-01-03T00:10:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.