Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework

• URL: http://arxiv.org/abs/2602.19549v1
• Date: Mon, 23 Feb 2026 06:45:19 GMT
• Title: Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework
• Authors: Yibo Yan, Mingdong Ou, Yi Cao, Xin Zou, Jiahao Huo, Shuliang Liu, James Kwok, Xuming Hu,
• Abstract summary: Visual Document Retrieval (VDR) aims to retrieve relevant pages within vast corpora of visually-rich documents.<n>Current efficiency methods like pruning and merging address imperfectly, creating a difficult trade-off between compression rate and feature fidelity.<n>We introduce Prune-then-Merge, a novel two-stage framework that synergizes these complementary approaches.
• Score: 39.59931739606983
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Document Retrieval (VDR), which aims to retrieve relevant pages within vast corpora of visually-rich documents, is of significance in current multimodal retrieval applications. The state-of-the-art multi-vector paradigm excels in performance but suffers from prohibitive overhead, a problem that current efficiency methods like pruning and merging address imperfectly, creating a difficult trade-off between compression rate and feature fidelity. To overcome this dilemma, we introduce Prune-then-Merge, a novel two-stage framework that synergizes these complementary approaches. Our method first employs an adaptive pruning stage to filter out low-information patches, creating a refined, high-signal set of embeddings. Subsequently, a hierarchical merging stage compresses this pre-filtered set, effectively summarizing semantic content without the noise-induced feature dilution seen in single-stage methods. Extensive experiments on 29 VDR datasets demonstrate that our framework consistently outperforms existing methods, significantly extending the near-lossless compression range and providing robust performance at high compression ratios.

Related papers

• Multivector Reranking in the Era of Strong First-Stage Retrievers [11.098422338598454]
  We reproduce several state-of-the-art multivector retrieval methods on two publicly available datasets.<n>We show that replacing the token-level gather phase with a single-vector document retriever produces a smaller and more semantically coherent candidate set.<n>Our two-stage approach achieves over $24times$ speedup over the state-of-the-art multivector retrieval systems, while maintaining comparable or superior retrieval quality.
  arXiv  Detail & Related papers  (2026-01-08T18:22:18Z)
• Rethinking Autoregressive Models for Lossless Image Compression via Hierarchical Parallelism and Progressive Adaptation [75.58269386927076]
  Autoregressive (AR) models are often dismissed as impractical due to prohibitive computational cost.<n>This work re-thinks this paradigm, introducing a framework built on hierarchical parallelism and progressive adaptation.<n> Experiments on diverse datasets (natural, satellite, medical) validate that our method achieves new state-of-the-art compression.
  arXiv  Detail & Related papers  (2025-11-14T06:27:58Z)
• Modest-Align: Data-Efficient Alignment for Vision-Language Models [67.48633659305592]
  Cross-modal alignment models often suffer from overconfidence and degraded performance when operating in resource-constrained settings.<n>We propose Modest-Align, a lightweight alignment framework designed for robustness and efficiency.<n>Our method offers a practical and scalable solution for cross-modal alignment in real-world, low-resource scenarios.
  arXiv  Detail & Related papers  (2025-10-24T16:11:10Z)
• Diverse Text-to-Image Generation via Contrastive Noise Optimization [60.48914865049489]
  Text-to-image (T2I) diffusion models have demonstrated impressive performance in generating high-fidelity images.<n>Existing approaches typically optimize intermediate latents or text conditions during inference.<n>We introduce Contrastive Noise Optimization, a simple yet effective method that addresses the diversity issue from a distinct perspective.
  arXiv  Detail & Related papers  (2025-10-04T13:51:32Z)
• FastFit: Accelerating Multi-Reference Virtual Try-On via Cacheable Diffusion Models [59.8871829077739]
  FastFit is a high-speed multi-reference virtual try-on framework based on a novel cacheable diffusion architecture.<n>Our model fully decouples reference feature encoding from the denoising process with negligible parameter overhead.<n>This allows reference features to be computed only once and losslessly reused across all steps.
  arXiv  Detail & Related papers  (2025-08-28T09:25:52Z)
• Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval [11.20814404187967]
  We propose a framework that enables bi-directional learning between dense and sparse representations through Self-Knowledge Distillation.<n>This bi-directional learning is achieved using an integrated similarity score-a weighted sum of dense and sparse similarities-which serves as a shared teacher signal for both representations.<n>Experiments on MSCOCO and Flickr30k demonstrate that our sparse retriever not only outperforms existing sparse baselines, but also achieves performance comparable to-or even surpassing-its dense counterparts.
  arXiv  Detail & Related papers  (2025-08-22T13:25:58Z)
• Generalized Correspondence Matching via Flexible Hierarchical Refinement and Patch Descriptor Distillation [13.802788788420175]
  Correspondence matching plays a crucial role in numerous robotics applications. This paper addresses the limitations of deep feature matching (DFM), a state-of-the-art (SoTA) plug-and-play correspondence matching approach. Our proposed method achieves an overall performance in terms of mean matching accuracy of 0.68, 0.92, and 0.95 with respect to the tolerances of 1, 3, and 5 pixels, respectively.
  arXiv  Detail & Related papers  (2024-03-08T15:32:18Z)
• DocDiff: Document Enhancement via Residual Diffusion Models [7.972081359533047]
  We propose DocDiff, a diffusion-based framework specifically designed for document enhancement problems. DocDiff consists of two modules: the Coarse Predictor (CP) and the High-Frequency Residual Refinement (HRR) module. Our proposed HRR module in pre-trained DocDiff is plug-and-play and ready-to-use, with only 4.17M parameters.
  arXiv  Detail & Related papers  (2023-05-06T01:41:10Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.