Towards Lossless Token Pruning in Late-Interaction Retrieval Models

• URL: http://arxiv.org/abs/2504.12778v1
• Date: Thu, 17 Apr 2025 09:18:58 GMT
• Title: Towards Lossless Token Pruning in Late-Interaction Retrieval Models
• Abstract summary: Late interaction neural IR models like ColBERT offer a competitive effectiveness-efficiency trade-off across many benchmarks.<n>They require a huge memory space to store the contextual representation for all the document tokens.<n>We propose a principled approach to define how to prune tokens without impacting the score between a document and a query.
• Score: 10.983837305643723
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Late interaction neural IR models like ColBERT offer a competitive effectiveness-efficiency trade-off across many benchmarks. However, they require a huge memory space to store the contextual representation for all the document tokens. Some works have proposed using either heuristics or statistical-based techniques to prune tokens from each document. This however doesn't guarantee that the removed tokens have no impact on the retrieval score. Our work uses a principled approach to define how to prune tokens without impacting the score between a document and a query. We introduce three regularization losses, that induce a solution with high pruning ratios, as well as two pruning strategies. We study them experimentally (in and out-domain), showing that we can preserve ColBERT's performance while using only 30\% of the tokens.

Related papers

• LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers [76.59130257385826]
  Intermediate merge residues in BPE vocabularies are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage.<n>We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens.<n>Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.
  arXiv  Detail & Related papers  (2026-02-04T16:19:05Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention [50.97683288777336]
  Multimodal Large Language Models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens.<n>Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention.<n>We propose HoloV, a plug-and-play visual token pruning framework for efficient inference.
  arXiv  Detail & Related papers  (2025-10-03T11:33:40Z)
• TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models [4.779482139419908]
  We introduce a mutual information-based token pruning strategy that removes visual tokens semantically with textual tokens.<n>Our method maintains strong performance while reducing textual tokens by 88.9% on models such as LLaVA-15-7B and LLaVA--7B.
  arXiv  Detail & Related papers  (2025-08-30T02:43:50Z)
• Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers [11.916258576313776]
  Vision Transformer (ViT) has achieved impressive results across various vision tasks.<n>Recent methods have aimed to reduce ViT's $O(n2)$ complexity by pruning unimportant tokens.<n>We introduce a novel bf Block-based Symmetric Pruning and Fusion for efficient ViT.
  arXiv  Detail & Related papers  (2025-07-16T10:48:56Z)
• Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings [70.26204343623215]
  ColPali/ColQwen2 encodes each page into multiple patch-level embeddings and leads to excessive memory usage.<n>This empirical study investigates methods to reduce patch embeddings per page at minimum performance degradation.
  arXiv  Detail & Related papers  (2025-06-05T13:06:01Z)
• Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More [18.928285521147057]
  We show that importance is not an ideal indicator to decide whether a token should be pruned. We propose DART (Duplication-Aware Reduction of Tokens), which prunes tokens based on its duplication with other tokens. Experiments demonstrate that DART can prune 88.9% vision tokens while maintaining comparable performance.
  arXiv  Detail & Related papers  (2025-02-17T06:56:28Z)
• ElasticTok: Adaptive Tokenization for Image and Video [109.75935878130582]
  We introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens. During inference, ElasticTok can dynamically allocate tokens when needed. Our evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage.
  arXiv  Detail & Related papers  (2024-10-10T20:54:15Z)
• Object Recognition as Next Token Prediction [99.40793702627396]
  We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels.
  arXiv  Detail & Related papers  (2023-12-04T18:58:40Z)
• Tree Cross Attention [59.8891512435847]
  Tree Cross Attention (TCA) is a module based on Cross Attention that only retrieves information from a logarithmic $mathcalO(log(N))$ number of tokens for performing inference. We show that TCA performs comparable to Cross Attention across various classification and uncertainty regression tasks while being significantly more token-efficient.
  arXiv  Detail & Related papers  (2023-09-29T16:50:23Z)
• Extract-and-Adaptation Network for 3D Interacting Hand Mesh Recovery [64.37035857740781]
  We present EANet, extract-and-adaptation network, with EABlock, the main component of our network. Our two novel tokens are from a combination of separated two hand features; hence, it is much more robust to the distant token problem. The proposed EANet achieves the state-of-the-art performance on 3D interacting hands benchmarks.
  arXiv  Detail & Related papers  (2023-09-05T04:18:03Z)
• Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation [89.88214896713846]
  STA score considers two critical factors: temporal redundancy and semantic importance. We apply the STA module to off-the-shelf video Transformers and Videowins. Results: Kinetics-400 and Something-Something V2 achieve 30% overshelf reduction with a negligible 0.2% accuracy drop.
  arXiv  Detail & Related papers  (2023-08-08T19:38:15Z)
• Revisiting Token Pruning for Object Detection and Instance Segmentation [25.3324628669201]
  We investigate token pruning to accelerate inference for object and instance segmentation. We show a reduction in performance decline from 1.5 mAP to 0.3 mAP in both boxes and masks, compared to existing token pruning methods.
  arXiv  Detail & Related papers  (2023-06-12T11:55:33Z)
• Multi-Scale And Token Mergence: Make Your ViT More Efficient [3.087140219508349]
  Vision Transformer (ViT) has emerged as a prevalent model in the computer vision domain. We propose a novel token pruning method that retains information from non-crucial tokens by merging them with more crucial tokens. Our method achieves a remarkable 33% reduction in computational costs while only incurring a 0.1% decrease in accuracy on DeiT-S.
  arXiv  Detail & Related papers  (2023-06-08T02:58:15Z)
• CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval [72.90850213615427]
  Multi-vector retrieval methods combine the merits of sparse (e.g. BM25) and dense (e.g. DPR) retrievers. These methods are orders of magnitude slower and need much more space to store their indices compared to their single-vector counterparts. We propose conditional token interaction via dynamic lexical routing, namely CITADEL, for efficient and effective multi-vector retrieval.
  arXiv  Detail & Related papers  (2022-11-18T18:27:35Z)
• Breaking BERT: Evaluating and Optimizing Sparsified Attention [13.529939025511242]
  We evaluate the impact of sparsification patterns with a series of ablation experiments. We find that even using attention that is at least 78% sparse can have little effect on performance if applied at later transformer layers.
  arXiv  Detail & Related papers  (2022-10-07T22:32:27Z)
• AdapLeR: Speeding up Inference by Adaptive Length Reduction [15.57872065467772]
  We propose a novel approach for reducing the computational cost of BERT with minimal loss in downstream performance. Our method dynamically eliminates less contributing tokens through layers, resulting in shorter lengths and consequently lower computational cost. Our experiments on several diverse classification tasks show speedups up to 22x during inference time without much sacrifice in performance.
  arXiv  Detail & Related papers  (2022-03-16T23:41:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.