Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models

• URL: http://arxiv.org/abs/2605.11939v1
• Date: Tue, 12 May 2026 10:50:43 GMT
• Title: Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models
• Abstract summary: We propose cluster-aware neural collapse prompt tuning (CPT)<n>CPT enhances the discriminability of tail classes in prompt-tuned vision-language models without sacrificing generalization.<n>Experiments on 11 diverse datasets demonstrate that CPT outperforms SOTA methods.
• Score: 20.68004586002534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt learning has emerged as an efficient alternative to fine-tuning pre-trained vision-language models (VLMs). Despite its promise, current methods still struggle to maintain tail-class discriminability when adapting to class-imbalanced datasets. In this work, we propose cluster-aware neural collapse prompt tuning (CPT), which enhances the discriminability of tail classes in prompt-tuned VLMs without sacrificing their overall generalization. First, we design a cluster-invariant space by mining semantic assignments from the pre-trained VLM and mapping them to prompt-tuned features. This computes cluster-level boundaries and restricts the constraints to local neighborhoods, which reduces interference with the global semantic structure of the pre-trained VLM. Second, we introduce neural-collapse-driven discriminability optimization with three losses: textual Equiangular Tight Frame (ETF) separation loss, class-wise convergence loss, and rotation stabilization loss. These losses work together to shape intra-cluster geometry for better inter-class separation and intra-class alignment. Extensive experiments on 11 diverse datasets demonstrate that CPT outperforms SOTA methods, with stronger performance on long-tail classes and good generalization to unseen classes.

Related papers

• Silhouette Loss: Differentiable Global Structure Learning for Deep Representations [0.0]
  We introduce Soft Silhouette Loss, a novel differentiable objective inspired by the classical silhouette coefficient from clustering analysis.<n>Our formulation evaluates each sample against all classes in the batch, providing a batch-level notion of global structure.<n>It can be seamlessly combined with cross-entropy, and is also complementary to supervised contrastive learning.
  arXiv  Detail & Related papers  (2026-03-27T12:11:47Z)
• Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score [11.74414842618874]
  We show that modeling fine-grained cross-modal interactions during adaptation produces more accurate, class-discriminative pseudo-labels.<n>We introduce Fine-grained Alignment and Interaction Refinement (FAIR), an innovative approach that dynamically aligns localized image features with descriptive language embeddings.<n>Our approach, FAIR, delivers a substantial performance boost in fine-grained unsupervised adaptation, achieving a notable overall gain of 2.78%.
  arXiv  Detail & Related papers  (2025-07-13T12:38:38Z)
• Unbiased Max-Min Embedding Classification for Transductive Few-Shot Learning: Clustering and Classification Are All You Need [83.10178754323955]
  Few-shot learning enables models to generalize from only a few labeled examples.<n>We propose the Unbiased Max-Min Embedding Classification (UMMEC) Method, which addresses the key challenges in few-shot learning.<n>Our method significantly improves classification performance with minimal labeled data, advancing the state-of-the-art in annotatedL.
  arXiv  Detail & Related papers  (2025-03-28T07:23:07Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Class-Imbalanced Semi-Supervised Learning for Large-Scale Point Cloud Semantic Segmentation via Decoupling Optimization [64.36097398869774]
  Semi-supervised learning (SSL) has been an active research topic for large-scale 3D scene understanding. The existing SSL-based methods suffer from severe training bias due to class imbalance and long-tail distributions of the point cloud data. We introduce a new decoupling optimization framework, which disentangles feature representation learning and classifier in an alternative optimization manner to shift the bias decision boundary effectively.
  arXiv  Detail & Related papers  (2024-01-13T04:16:40Z)
• Self-Supervised Class Incremental Learning [51.62542103481908]
  Existing Class Incremental Learning (CIL) methods are based on a supervised classification framework sensitive to data labels. When updating them based on the new class data, they suffer from catastrophic forgetting: the model cannot discern old class data clearly from the new. In this paper, we explore the performance of Self-Supervised representation learning in Class Incremental Learning (SSCIL) for the first time.
  arXiv  Detail & Related papers  (2021-11-18T06:58:19Z)
• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning [94.35586521144117]
  We investigate whether applying contrastive learning to fine-tuning would bring further benefits. We propose Contrast-regularized tuning (Core-tuning), a novel approach for fine-tuning contrastive self-supervised visual models.
  arXiv  Detail & Related papers  (2021-02-12T16:31:24Z)
• Generalized Zero-Shot Learning Via Over-Complete Distribution [79.5140590952889]
  We propose to generate an Over-Complete Distribution (OCD) using Conditional Variational Autoencoder (CVAE) of both seen and unseen classes. The effectiveness of the framework is evaluated using both Zero-Shot Learning and Generalized Zero-Shot Learning protocols.
  arXiv  Detail & Related papers  (2020-04-01T19:05:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.