CRISP: Clustering Multi-Vector Representations for Denoising and Pruning

• URL: http://arxiv.org/abs/2505.11471v1
• Date: Fri, 16 May 2025 17:26:16 GMT
• Title: CRISP: Clustering Multi-Vector Representations for Denoising and Pruning
• Authors: João Veneroso, Rajesh Jayaram, Jinmeng Rao, Gustavo Hernández Ábrego, Majid Hadian, Daniel Cer,
• Abstract summary: Multi-vector models such as ColBERT deliver state-of-the-art performance by representing queries and documents by contextualized token-level embeddings.<n>A common approach to mitigate this overhead is to cluster the model's frozen vectors, but this strategy's effectiveness is limited by the intrinsic clusterability of these embeddings.<n>We introduce CRISP, a novel multi-vector training method which learns inherently clusterable representations directly within the end-to-end training process.
• Score: 7.580000668015255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-vector models, such as ColBERT, are a significant advancement in neural information retrieval (IR), delivering state-of-the-art performance by representing queries and documents by multiple contextualized token-level embeddings. However, this increased representation size introduces considerable storage and computational overheads which have hindered widespread adoption in practice. A common approach to mitigate this overhead is to cluster the model's frozen vectors, but this strategy's effectiveness is fundamentally limited by the intrinsic clusterability of these embeddings. In this work, we introduce CRISP (Clustered Representations with Intrinsic Structure Pruning), a novel multi-vector training method which learns inherently clusterable representations directly within the end-to-end training process. By integrating clustering into the training phase rather than imposing it post-hoc, CRISP significantly outperforms post-hoc clustering at all representation sizes, as well as other token pruning methods. On the BEIR retrieval benchmarks, CRISP achieves a significant rate of ~3x reduction in the number of vectors while outperforming the original unpruned model. This indicates that learned clustering effectively denoises the model by filtering irrelevant information, thereby generating more robust multi-vector representations. With more aggressive clustering, CRISP achieves an 11x reduction in the number of vectors with only a $3.6\%$ quality loss.

Related papers

• Towards Learnable Anchor for Deep Multi-View Clustering [49.767879678193005]
  In this paper, we propose the Deep Multi-view Anchor Clustering (DMAC) model that performs clustering in linear time.<n>With the optimal anchors, the full sample graph is calculated to derive a discriminative embedding for clustering.<n>Experiments on several datasets demonstrate superior performance and efficiency of DMAC compared to state-of-the-art competitors.
  arXiv  Detail & Related papers  (2025-03-16T09:38:11Z)
• Cluster Specific Representation Learning [1.6727186769396276]
  Despite its widespread application, there is no established definition of a good'' representation.<n>We propose a downstream-agnostic formulation: when inherent clusters exist in the data, the representations should be specific to each cluster.<n>Under this idea, we develop a meta-algorithm that jointly learns cluster-specific representations and cluster assignments.
  arXiv  Detail & Related papers  (2024-12-04T16:59:37Z)
• Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens [57.37893387775829]
  We introduce a fast and balanced clustering method, named textbfSemantic textbfEquitable textbfClustering (SEC) SEC clusters tokens based on their global semantic relevance in an efficient, straightforward manner. We propose a versatile vision backbone, SECViT, to serve as a vision language connector.
  arXiv  Detail & Related papers  (2024-05-22T04:49:00Z)
• GCC: Generative Calibration Clustering [55.44944397168619]
  We propose a novel Generative Clustering (GCC) method to incorporate feature learning and augmentation into clustering procedure. First, we develop a discrimirative feature alignment mechanism to discover intrinsic relationship across real and generated samples. Second, we design a self-supervised metric learning to generate more reliable cluster assignment.
  arXiv  Detail & Related papers  (2024-04-14T01:51:11Z)
• End-to-end Learnable Clustering for Intent Learning in Recommendation [54.157784572994316]
  We propose a novel intent learning method termed underlineELCRec. It unifies behavior representation learning into an underlineEnd-to-end underlineLearnable underlineClustering framework. We deploy this method on the industrial recommendation system with 130 million page views and achieve promising results.
  arXiv  Detail & Related papers  (2024-01-11T15:22:55Z)
• CLC: Cluster Assignment via Contrastive Representation Learning [9.631532215759256]
  We propose Contrastive Learning-based Clustering (CLC), which uses contrastive learning to directly learn cluster assignment. We achieve 53.4% accuracy on the full ImageNet dataset and outperform existing methods by large margins.
  arXiv  Detail & Related papers  (2023-06-08T07:15:13Z)
• Large-Margin Representation Learning for Texture Classification [67.94823375350433]
  This paper presents a novel approach combining convolutional layers (CLs) and large-margin metric learning for training supervised models on small datasets for texture classification. The experimental results on texture and histopathologic image datasets have shown that the proposed approach achieves competitive accuracy with lower computational cost and faster convergence when compared to equivalent CNNs.
  arXiv  Detail & Related papers  (2022-06-17T04:07:45Z)
• Exploring Non-Contrastive Representation Learning for Deep Clustering [23.546602131801205]
  Non-contrastive representation learning for deep clustering, termed NCC, is based on BYOL, a representative method without negative examples. NCC forms an embedding space where all clusters are well-separated and within-cluster examples are compact. Experimental results on several clustering benchmark datasets including ImageNet-1K demonstrate that NCC outperforms the state-of-the-art methods by a significant margin.
  arXiv  Detail & Related papers  (2021-11-23T12:21:53Z)
• Meta Clustering Learning for Large-scale Unsupervised Person Re-identification [124.54749810371986]
  We propose a "small data for big task" paradigm dubbed Meta Clustering Learning (MCL) MCL only pseudo-labels a subset of the entire unlabeled data via clustering to save computing for the first-phase training. Our method significantly saves computational cost while achieving a comparable or even better performance compared to prior works.
  arXiv  Detail & Related papers  (2021-11-19T04:10:18Z)
• Learning Statistical Representation with Joint Deep Embedded Clustering [2.1267423178232407]
  StatDEC is an unsupervised framework for joint statistical representation learning and clustering. Our experiments show that using these representations, one can considerably improve results on imbalanced image clustering across a variety of image datasets.
  arXiv  Detail & Related papers  (2021-09-11T09:26:52Z)
• Image Clustering using an Augmented Generative Adversarial Network and Information Maximization [9.614694312155798]
  We propose a deep clustering framework consisting of a modified generative adversarial network (GAN) and an auxiliary classifier. The proposed method significantly outperforms state-of-the-art clustering methods on CIFAR-10 and CIFAR-100, and is competitive on the STL10 and MNIST datasets.
  arXiv  Detail & Related papers  (2020-11-08T22:20:33Z)
• Generalized Zero-Shot Learning Via Over-Complete Distribution [79.5140590952889]
  We propose to generate an Over-Complete Distribution (OCD) using Conditional Variational Autoencoder (CVAE) of both seen and unseen classes. The effectiveness of the framework is evaluated using both Zero-Shot Learning and Generalized Zero-Shot Learning protocols.
  arXiv  Detail & Related papers  (2020-04-01T19:05:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.