Look in the Middle: Structural Anchor Pruning for Scalable Visual RAG Indexing

• URL: http://arxiv.org/abs/2601.20107v1
• Date: Tue, 27 Jan 2026 22:50:11 GMT
• Title: Look in the Middle: Structural Anchor Pruning for Scalable Visual RAG Indexing
• Abstract summary: Recent Vision-Language Models (e.g., ColPali) enable fine-grained Visual Document Retrieval (VDR) but incur prohibitive index vector size overheads.<n>We propose Structural Anchor Pruning (SAP), a training-free pruning method that identifies key visual patches from middle layers to achieve high performance compression.
• Score: 11.750217913598057
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent Vision-Language Models (e.g., ColPali) enable fine-grained Visual Document Retrieval (VDR) but incur prohibitive index vector size overheads. Training-free pruning solutions (e.g., EOS-attention based methods) can reduce index vector size by approximately 60% without model adaptation, but often underperform random selection in high-compression scenarios (> 80%). Prior research (e.g., Light-ColPali) attributes this to the conclusion that visual token importance is inherently query-dependent, thereby questioning the feasibility of training-free pruning. In this work, we propose Structural Anchor Pruning (SAP), a training-free pruning method that identifies key visual patches from middle layers to achieve high performance compression. We also introduce Oracle Score Retention (OSR) protocol to evaluate how layer-wise information affects compression efficiency. Evaluations on the ViDoRe benchmark demonstrate that SAP reduces index vectors by over 90% while maintaining robust retrieval fidelity, providing a highly scalable solution for Visual RAG. Furthermore, our OSR-based analysis reveals that semantic structural anchor patches persist in the middle layers, unlike traditional pruning solutions that focus on the final layer where structural signals dissipate.

Related papers

• Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation [42.92624141712034]
  Prompted Information Bottlenecks (PIB) is a framework that regularizes layer-wise compression-sufficiency trade-offs and promotes a more coherent cross-layer information path.<n>PIB achieves strong performance across 34 datasets, reaching 92.1% on FGVC, 93.01% on HTA, and 77.33% on VTAB-1k.<n>These results position PIB as both a practical method and an information-allocation perspective for adapting frozen vision foundation models.
  arXiv  Detail & Related papers  (2026-07-24T04:39:55Z)
• Mind the Gap: Disentangling Performance Bottlenecks in Video Instance Segmentation [0.34410212782758043]
  Video Instance (VIS) classification, segmentation, and tracking objectives are jointly evaluated.<n>We introduce a diagnostic framework that formulates identity and class assignment as an algorithmic Linear Program (ILP)<n>We also introduce TrackLens, a visual tool that translates magnitude into observable, query-level failure modes.
  arXiv  Detail & Related papers  (2026-06-05T15:32:48Z)
• No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval [51.43543998583709]
  Single-stage Sparse Retrieval (SSR) is a paradigm shift that replaces expensive clustering with efficient sparse coding.<n>It reduces indexing time by 15x compared to ColBERTv2, halves retrieval latency, and simultaneously improves retrieval performance.
  arXiv  Detail & Related papers  (2026-05-28T15:53:34Z)
• Faster or Stronger: Towards Flexible Visual Place Recognition via Weighted Aggregation and Token Pruning [27.4981354426677]
  Visual Place Recognition (VPR) aims to match a query image to reference images of the same place in a large-scale database.<n>We introduce WeiToP, a VPR-oriented token pruning framework that reduces feature extraction cost via self-distillation.<n>WeiToP enables plug-and-play token pruning at inference time, allowing flexible and on-demand control over the accuracy-efficiency trade-off without additional training.
  arXiv  Detail & Related papers  (2026-05-19T23:01:57Z)
• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• DepthCache: Depth-Guided Training-Free Visual Token Merging for Vision-Language-Action Model Inference [5.305950698447464]
  Vision-Language-Action (VLA) models enable generalist robotic manipulation but suffer from high inference latency.<n>We present DepthCache, a training-free framework that leverages depth as a structural prior for visual token compression.<n>On the LIBERO benchmark, DepthCache achieves up to 1.28x inference speedup with less than 1% average success rate degradation.
  arXiv  Detail & Related papers  (2026-03-11T06:40:44Z)
• Multi-Vector Index Compression in Any Modality [73.7330345057813]
  Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos.<n>We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC)<n>AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation.
  arXiv  Detail & Related papers  (2026-02-24T18:57:33Z)
• ForgeryVCR: Visual-Centric Reasoning via Efficient Forensic Tools in MLLMs for Image Forgery Detection and Localization [62.03035862528452]
  ForgeryVCR is a framework that materializes imperceptible traces into explicit visual intermediates via Visual-Centric Reasoning.<n>ForgeryVCR achieves state-of-the-art (SOTA) performance in both detection and localization tasks.
  arXiv  Detail & Related papers  (2026-02-15T11:14:47Z)
• SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs [34.330422211810685]
  We introduce (Separating Perception And Reasoning Circuits) a modular framework that explicitly decouples visual perception from reasoning.<n>Inspired by sequential sensory-to-cognitive processing in the brain, it implements a two-stage pipeline where the model first performs explicit visual search to localize question-relevant regions, then conditions its reasoning on those regions to produce the final answer.<n>Across challenging visual reasoning benchmarks, it outperforms monolithic baselines and strong visual-grounding approaches.
  arXiv  Detail & Related papers  (2026-02-06T10:05:25Z)
• Unleashing Degradation-Carrying Features in Symmetric U-Net: Simpler and Stronger Baselines for All-in-One Image Restoration [52.82397287366076]
  All-in-one image restoration aims to handle diverse degradations (e.g., noise, blur, adverse weather) within a unified framework.<n>In this work, we reveal a critical insight: well-crafted feature extraction inherently encodes degradation-carrying information.<n>Our symmetric design preserves intrinsic degradation signals robustly, rendering simple additive fusion in skip connections.
  arXiv  Detail & Related papers  (2025-12-11T12:20:31Z)
• A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Models [94.49953824684853]
  We introduce a dynamic pruning framework, GlimpsePrune, inspired by human cognition.<n>It takes a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single forward pass before answer generation.<n>An enhanced GlimpsePrune+ achieves 110% of the baseline performance while maintaining a similarly high pruning rate.
  arXiv  Detail & Related papers  (2025-08-03T02:15:43Z)
• TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generation [48.61855865678161]
  We present the first general test-time scaling framework for visual auto-regressive ( VAR) models.<n>We propose clustering-based diversity search and resampling-based potential selection.<n>Experiments on the powerful VAR model Infinity show a notable 8.7% GenEval score improvement.
  arXiv  Detail & Related papers  (2025-07-24T16:04:55Z)
• High-Layer Attention Pruning with Rescaling [14.141903038286362]
  Pruning is a highly effective approach for compressing large language models (LLMs)<n>We propose a novel pruning algorithm that strategically prunes attention heads in the model's higher layers.<n>We conduct comprehensive experiments on a wide range of LLMs, including LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B.
  arXiv  Detail & Related papers  (2025-07-02T17:15:05Z)
• Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuning [13.384550074613717]
  Visual Prompt Tuning is noted for its superior performance compared to full fine-tuning.<n>Ship significantly improves performance, achieving a 4.9% gain in accuracy over VPT with a ViT-B/16 backbone on VTAB-1k tasks.
  arXiv  Detail & Related papers  (2024-12-22T10:28:52Z)
• Pruning All-Rounder: Rethinking and Improving Inference Efficiency for Large Vision Language Models [42.124670377223175]
  We propose a novel framework for inference acceleration called the Pruning All-Rounder (PAR)<n>With a self-supervised learning manner, PAR achieves a superior balance between performance and efficiency. Notably, PAR is highly flexible, offering multiple pruning versions to address a range of acceleration scenarios.
  arXiv  Detail & Related papers  (2024-12-09T13:02:35Z)
• Mind the Backbone: Minimizing Backbone Distortion for Robust Object Detection [52.355018626115346]
  Building object detectors that are robust to domain shifts is critical for real-world applications. We propose to use Relative Gradient Norm as a way to measure the vulnerability of a backbone to feature distortion. We present recipes to boost OOD robustness for both types of backbones.
  arXiv  Detail & Related papers  (2023-03-26T14:50:43Z)
• COPS: Controlled Pruning Before Training Starts [68.8204255655161]
  State-of-the-art deep neural network (DNN) pruning techniques, applied one-shot before training starts, evaluate sparse architectures with the help of a single criterion -- called pruning score. In this work we do not concentrate on a single pruning criterion, but provide a framework for combining arbitrary GSSs to create more powerful pruning strategies.
  arXiv  Detail & Related papers  (2021-07-27T08:48:01Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.